report.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UGopYmmGkxFY"
      },
      "source": [
        "# Deep Learning Assignment 2023\n",
        "## From words to bounding boxes: exploring visual grounding using CLIP\n",
        "\n",
        "|     #    |                 |                                 @ |\n",
        "|:--------:|-----------------|----------------------------------:|\n",
        "| `238746` | Luca Mosetti    | luca.mosetti-1@studenti.unitn.it  |\n",
        "| `240074` | Stefano Genetti | stefano.genetti@studenti.unitn.it |\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5RA_k6W7qYv9"
      },
      "outputs": [],
      "source": [
        "%%shell\n",
        "if ! [ -d assets ]; then\n",
        "  gdown -q 1WTQNojr6KvWbzowuqfBDj5z4CYd03eeQ &&\n",
        "  tar --warning=no-unknown-keyword -xf assets.tar &&\n",
        "  rm assets.tar\n",
        "fi"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jf3_sh3snQ9k"
      },
      "source": [
        "## 1 Abstract"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g4J8jcsr7RK5"
      },
      "source": [
        "Visual grounding involves linking language and perception by grounding linguistic symbols in the visual world. More in depth, in this work we face the problem usually referred to by the literature as *Referring expression comprehension* (REC). In this context the overall goal is to localize a target object in an image described by a referring expression phrased in natural language. In order to accomplish this challenging task we rely on the CLIP (*Contrastive Language-Image Pre-training*) [2] pre-trained model as a starting point for transfer learning. The capabilities of this foundation model pose a starting point to design a joint embedding approach to solve the problem at hand. In this report we provide an overview of the strategies which we have adopted in order to fine-tune CLIP for the task under discussion. We have evaluated our proposed models on the commonly used RefCOCOg dataset [3]. In addition to this, our contribution is to provide three useful instances of the dataset filled with the bounding boxes proposed by some well known  object detection algorithms. As further explained in the following of this report this solution allows to considerably speed up the training procedure. We conveniently  provide these datasets together with the code to generate them at the following GitHub repository: https://github.com/StefanoGenettiUniTN/refcocog-augmentation. Furthermore, in the present notebook we alternate the text cells with code cells incorporating the implemented code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZtAevOMyFOcd"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/01.png)\n",
        "> \n",
        "> **Figure 1**\n",
        "> \n",
        "> *two woman one in black eatting and the other has a white shirt at the desk*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_0NsKknFU-F"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/02.png)\n",
        "> \n",
        "> **Figure 2**\n",
        "> \n",
        "> *a brown bull in front of feeding tub*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-rGXRkdbFbCb"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/03.png)\n",
        "> \n",
        "> **Figure 3**\n",
        "> \n",
        "> *green color vegetable in between potato and carrot*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_OvWZapnagY"
      },
      "source": [
        "## 2 Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wjKW65fKH2CF"
      },
      "source": [
        "Language and vision are closely related in daily life. We naturally use verbal descriptions in our conversations to refer to the objects in a given scene. Although such an activity is straightforward for the human being, the task of referring expression comprehension remains challenging for a software agent which has to bridge computer vision and natural language processing in order to achieve a comprehensive understanding of complex language semantics and various types of visual information. The problem has been receiving increasing attention from both academia and industry due to its great potential in vision-and-language navigation [1] and natural human-computer interaction. The aim of our work is to train a model which takes as input an image and a natural language prompt and outputs a single bounding box which corresponds to the entity referred to in the textual description (Figure 4).\n",
        "\n",
        "\\begin{equation}\n",
        "    f: I \\times P \\rightarrow B\n",
        "\\end{equation}\n",
        "\n",
        "According to the literature, the most common methods to tackle the task at hand are based on the encoding of image regions and expressions into the same vector space [4]. To this end we adopt the CLIP (Contrastive Language-Image Pre-training) [2] model as a foundation model for our framework. This strategy allows us to take advantage of a powerful model pre-trained with massive computational resources and consequently reduce the amount of power that we need to obtain meaningful results. Clearly, visual grounding is not the original purpose of CLIP. Consequently, we need to perform transfer learning in order to fine-tune the original model to build a customized one that excels in our downstream task. In this paper we provide a detailed overview of the solutions that we have studied to solve the problem. To this end we have trained and evaluated several model architectures according to the metrics commonly suggested by the literature, on the RefCOCOg dataset [3], a variant of the Referring Expression Generation (REG) dataset, which is particularly suitable in our case. For each implemented model we report the obtained performances bringing to light its strengths and weaknesses. A methodological comparison of the proposed architecture designs has allowed us to select the most promising implementation. As outlined at the end of this document, the final model has been further refined in an attempt to improve its generalization capabilities. The overall goal of this work is not to achieve state of the art performances. Rather, our contribution is to suggest original solutions to tackle the problem and highlight promising directions which should be further investigated with stronger hardware capabilities. In this regard, throughout the report we strain our attention on several valuable strategies which have been adopted in order to deal with limited time and computational resources. The notebook is organized as follows.\n",
        "*   At the beginning we provide a brief overview of the related works proposed by the literature over the years to face similar challenges.\n",
        "*   Then, we explain the peculiarities of our reference dataset. There is no predefined dataset class appropriate for the visual grounding task. Hence, we describe how we create our custom dataset classes to load and read the available data collection correctly and appropriately.\n",
        "* In Section 6 we describe the metrics which we have adopted in order to evaluate and compare the implemented solutions.\n",
        "* In the subsequent section we describe our training free baseline algorithm which has represented a convenient starting point for our project.\n",
        "* In the following sections we comment on the architectures that we have designed on top of the observations and experience progressively maturated. First of all we illustrate in Section 9 the object detection algorithms which have been examined in order to maximize the quality of the proposed bounding boxes to be evaluated.\n",
        "* In Section 10 we describe our first fine-tuned architecture. In this first implementation we focus on a standard fine-tuning approach.\n",
        "* Subsequently, inspired by the work of Sachin Goyal et al. [5] we have tried to fine-tune our model following the same training procedure employed by CLIP.\n",
        "* In Section 12 we propose an original technique to exploit the self-attention mechanism in order to produce contextualized latent space representations of visual and textual prompts.\n",
        "* This pipeline allowed us to identify the most auspicious model among the ones considered. In Section 13 we present our solutions to further improve its generalization capabilities.\n",
        "* At the end, we conclude the report with some final considerations and valuable suggestions to inspire further research on the field.     \n",
        "\n",
        "In between the textual descriptions of our findings we conveniently provide our implemented code. Everything has been written in Python programming language and specifically with the PyTorch machine learning framework. We strain our attention to be as clear as possible with our lines of code. To this end we have carefully annotated the data types which should make the overall implementation more comprehensible for the reader. To achieve this we have used Python [typing](https://docs.python.org/3/library/typing.html) and [Pydantic](https://docs.pydantic.dev/latest/). Furthermore, we have sometimes utilized the [doctest](https://docs.python.org/3/library/doctest.html) module to verify that the implemented functions behave exactly as shown.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "USX6E9pCzIr0"
      },
      "source": [
        "> ![input-output.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/04.png)\n",
        "> \n",
        "> **Figure 4**\n",
        "> \n",
        "> On the left the input of our problem. On the right a bounding box is drawn around the most interesting portion of the image according to the description. As clarified in the following, sometimes in our dataset there are provided more than a single description for a given bounding box. In this case we profitably take advantage of the multiple prompts available trying to achieve better predictions.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pjWiXoYX38ph"
      },
      "source": [
        "## 3 Hardware"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDfjKe8P3_zw"
      },
      "source": [
        "In this section we briefly mention the hardware infrastructures that we have used to execute our experiments, train the proposed neural networks and evaluate our models. Throughout the project we have strained our attention to carefully plan our executions in order to deal with limited resources and to invest the computational time at our disposal as fruitfully as possible.\n",
        "\n",
        "The whole project has been written with the environment provided by the free of charge version of Google Colab. The platform allows to execute the code with only the CPU or with Nvidia T4 GPUs. Unfortunately, the execution on GPU is subject to strict and very limited time constraints. Consequently, we use the CPU mainly to verify that the implemented code works as intended and for debugging. Once the overall architecture behaves correctly, we perform short time tests with small dataset subsets on GPU with the aim of understanding whether the model is learning something and how its performance could be refined.\n",
        "\n",
        "In addition to this we had at our disposal 50 hours of execution on the very powerful GPUs Tesla V100 provided by Microsoft Azure. The performance evaluations presented in this report have been computed on models trained on this hardware for a reasonable amount of time.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B7DjZlVvwbM8"
      },
      "outputs": [],
      "source": [
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JrotexbA6n1G"
      },
      "outputs": [],
      "source": [
        "%%shell\n",
        "tee requirements.txt << END\n",
        "ftfy\n",
        "jaxtyping\n",
        "jupyter\n",
        "matplotlib\n",
        "optuna\n",
        "pandas\n",
        "pydantic\n",
        "regex\n",
        "sentencepiece\n",
        "tensorboard\n",
        "textaugment\n",
        "torch\n",
        "torchinfo\n",
        "torchvision\n",
        "tqdm\n",
        "transformers\n",
        "END\n",
        "\n",
        "pip install -q -r requirements.txt\n",
        "pip install -q git+https://github.com/openai/CLIP.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EUJ0Q3416lFn"
      },
      "outputs": [],
      "source": [
        "import csv\n",
        "import doctest\n",
        "import itertools as it\n",
        "import math\n",
        "import os\n",
        "import typing as t\n",
        "import random\n",
        "\n",
        "import clip\n",
        "import matplotlib.pyplot as plt\n",
        "import optuna\n",
        "import pandas as pd\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "from collections import defaultdict\n",
        "from clip.model import CLIP\n",
        "from jaxtyping import Float, UInt, Int\n",
        "from pydantic.dataclasses import dataclass\n",
        "from torch.utils.data import DataLoader, Dataset\n",
        "from torch.utils.tensorboard import SummaryWriter\n",
        "from torchinfo import summary\n",
        "from torchinfo.model_statistics import ModelStatistics\n",
        "from torchvision.io import read_image, ImageReadMode\n",
        "from torchvision.ops import box_iou, box_convert\n",
        "from torchvision.transforms import (\n",
        "    Compose,\n",
        "    Resize,\n",
        "    CenterCrop,\n",
        "    Normalize,\n",
        "    InterpolationMode,\n",
        "    ConvertImageDtype,\n",
        "    ColorJitter,\n",
        "    GaussianBlur,\n",
        "    RandomChoice,\n",
        "    RandomInvert,\n",
        "    RandomPosterize,\n",
        "    RandomSolarize,\n",
        "    RandomAdjustSharpness,\n",
        "    RandomAutocontrast,\n",
        "    RandomEqualize,\n",
        "    Grayscale,\n",
        ")\n",
        "from torchvision.transforms.functional import crop\n",
        "from tqdm.auto import tqdm, trange\n",
        "from optuna.study import Study\n",
        "from optuna.visualization import (\n",
        "    plot_contour,\n",
        "    plot_edf,\n",
        "    plot_intermediate_values,\n",
        "    plot_optimization_history,\n",
        "    plot_parallel_coordinate,\n",
        "    plot_param_importances,\n",
        "    plot_rank,\n",
        "    plot_slice,\n",
        "    plot_timeline,\n",
        ")\n",
        "from optuna.storages import RDBStorage\n",
        "from optuna.trial import Trial\n",
        "from optuna.study import Study"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ol8I22EE3J0D"
      },
      "outputs": [],
      "source": [
        "device: torch.device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "torch.set_default_device(device)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed_all(42)\n",
        "torch.use_deterministic_algorithms(False)  # CLIP uses non-deterministic algorithms\n",
        "g: torch.Generator = torch.Generator(device=device).manual_seed(42)\n",
        "random.seed(42)"
      ],
      "metadata": {
        "id": "GpNoXbU4cjPl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "11j0RuPSxB3R"
      },
      "outputs": [],
      "source": [
        "T = t.TypeVar(\"T\")\n",
        "K = t.TypeVar(\"K\")\n",
        "V = t.TypeVar(\"V\")\n",
        "\n",
        "\n",
        "def groupby(\n",
        "    xs: list[T],\n",
        "    map_key: t.Callable[[T], K],\n",
        "    map_value: t.Callable[[T], V] = lambda x: x,\n",
        ") -> dict[K, list[V]]:\n",
        "    return {\n",
        "        k: [map_value(v) for v in vs]\n",
        "        for k, vs in it.groupby(sorted(xs, key=map_key), key=map_key)\n",
        "    }\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def unzip(batch: list[tuple[T, ...]]) -> tuple[tuple[T, ...], ...]:\n",
        "    \"\"\"\n",
        "\n",
        "    >>> unzip([('A', 1), ('B', 2)])\n",
        "    (('A', 'B'), (1, 2))\n",
        "\n",
        "    \"\"\"\n",
        "    return tuple(zip(*batch))"
      ],
      "metadata": {
        "id": "GRZ8adBvc5w5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def best_bbox(\n",
        "    pred: Float[torch.Tensor, \"crops 4\"], groundtruth: Float[torch.Tensor, \"1 4\"]\n",
        ") -> int:\n",
        "    \"\"\"\n",
        "\n",
        "    >>> best_bbox(\n",
        "    ...     torch.tensor([[0, 0, 1, 1], [0, 0, 2, 2], [1, 1, 2, 2]]),\n",
        "    ...     torch.tensor([[0, 0, 1, 1]])\n",
        "    ... )\n",
        "    0\n",
        "\n",
        "    >>> best_bbox(\n",
        "    ...     torch.tensor([[0, 0, 0, 0], [0, 0, 2, 2], [1, 1, 2, 2]]),\n",
        "    ...     torch.tensor([[0, 0, 1, 1]])\n",
        "    ... )\n",
        "    1\n",
        "\n",
        "    \"\"\"\n",
        "    return torch.argmax(box_iou(pred, groundtruth)).item()"
      ],
      "metadata": {
        "id": "7jAqyxXLc7Nw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def eval_summary(model: nn.Module) -> ModelStatistics:\n",
        "    return summary(\n",
        "        model,\n",
        "        input_size=[(5, 3, 244, 244), (2, 77)],\n",
        "        dtypes=[torch.float, torch.int],\n",
        "        col_names=[\"input_size\", \"output_size\", \"num_params\", \"trainable\"],\n",
        "    )"
      ],
      "metadata": {
        "id": "Mb2xuA8cc8zE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def contrastive_summary(model: nn.Module) -> ModelStatistics:\n",
        "    return summary(\n",
        "        model,\n",
        "        input_size=[(8, 3, 244, 244), (8, 77)],\n",
        "        dtypes=[torch.float, torch.int],\n",
        "        col_names=[\"input_size\", \"output_size\", \"num_params\", \"trainable\"],\n",
        "    )"
      ],
      "metadata": {
        "id": "tc4e7p58c-IL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtVt3nQzn2Zs"
      },
      "source": [
        "## 4 Related work"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ddl18Yp38hqZ"
      },
      "source": [
        "To the best of our knowledge it is not obvious to understand whether our task falls into the realm of Referring Expression Comprehension or Visual Grounding problem. Actually the two categories are very similar. According to the exhaustive survey of Yanyuan Quiao, Chaorui Deng and Qi Wu [4], visual grounding is to localize multiple object regions in an image corresponding to multiple noun phrases from a sentence that describes the underlying scene. While the goal of referring expression comprehension is to find the best matching region by the given expression.\n",
        "More broadly, referring expression is normally associated with three tasks: generation, segmentation and comprehension.\n",
        "* Referring expression generation (REG) aims at generating a discriminative description of an object in an image, which is very similar to the image captioning task. Different from general image captions, referring expressions are more specific about an object or region in the image [4].\n",
        "* Referring expression segmentation (RES) aims to segment the referenced objects according to the referring expression [6].\n",
        "* Referring expression comprehension (REC) is the reverse task of REG, which aims at localizing objects in an image based on natural language descriptions. The REC problem is typically formulated as selecting the best region from a set of region proposals extracted from the image.\n",
        "\n",
        "Furthermore, even Object Detection resembles our objectives. However, although this latter uses predefined category labels to classify fixed objects, in our project we focus on natural language expressions to refer to objects. These phrases are more practical because they vary according to the content of images and texts, so they are more suitable for real application scenarios. Succeeding in this task is of crucial importance for other vision and language problems, such as Visual Question Answering [7][8] and Visual dialogue [9][10]. Though they have diverse model architectures, they necessitate a prior localization of the objects corresponding to a given language description or question. Notably, since the textual information is not a separate label, a simple detection method cannot meet the requirements.\n",
        "\n",
        "More in depth, the methods to face this problem proposed by the literature over the years are divided into seven categories: joint embedding approaches, modular-based approaches, graph-based approaches, approaches using external parsers, weakly supervised approaches, one stage approaches, vision-language pre-training approaches. In this work, we focus our attention on a joint embedding approach. In essence, the main idea behind these methods is to encode the image regions and the natural language prompts into the same vector space in order to link visual and textual representations. A representative and pioneered work in this field is the one proposed by Mao et al. [11]. In this case as depicted in Figure 5, they use a Convolutional Neural Network  to generate rich image representations by embedding input images into fixed-length vectors and a LSTM network to generate text features.\n",
        "\n",
        "By mimicking the core principles of previous joint embedding approaches, a crucial part of our solution is the representation of text and images in a shared latent space. To accomplish this, we rely on CLIP (Contrastive Language-Image Pre-training) [2], a recent large-scale model pretrained jointly on image and text data. More specifically, the architecture proposed by Radford et al., has been trained via a contrastive loss that finds a joint embedding over a set of paired image and text data. From other studies, this model has demonstrated exceptional performance on many downstream tasks. For instance, many works have fine-tuned the original model to perform zero-shot classification tasks [12][13]. In line with these approaches, in the following of the report we present our ideas to fine-tune CLIP for solving our downstream visual grounding problem.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rh4bhjEd2b9x"
      },
      "source": [
        "> ![formerApproach.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/05.png)\n",
        "> \n",
        "> **Figure 5**\n",
        "> \n",
        "> A common approach to solve visual grounding task using CNN-LSTM framework."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aP3dpLx7oMmB"
      },
      "source": [
        "## 5 Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqK6A0G2gy4Z"
      },
      "source": [
        "For the purpose of training and assessing the models presented in this report, we made use of the umd segment of the RefCOCOg dataset. This dataset contains an extensive assortment of 85'474 referring expressions, each corresponding to one of 54'822 unique objects present in 26'711 images.\n",
        "The aim of this section is to concisely describe the structure of the RefCOCOg dataset and the custom classes that we have prepared to load and read the dataset correctly.\n",
        "\n",
        "The RefCOCOg dataset was adapted to our needs by agglomerating and reducing the key information to 2 csv:\n",
        "\n",
        "- `refs.csv`\n",
        "- `sentences.csv`\n",
        "\n",
        "> `refs.csv`\n",
        ">\n",
        "> The single row refers to an image in the dataset and corresponds to a bounding box ground truth, expressed in `xyxy` coordinates\n",
        "\n",
        "> `sentences.csv`\n",
        ">\n",
        "> The single row refers to a bounding box in `refs.csv` and corresponds to a possible textual description of the scene enclosed therein\n",
        "\n",
        "In addition, as detailed in Section 9, the dataset has been enriched with 3 csv files:\n",
        "- `bboxes[YOLOv5].csv`\n",
        "- `bboxes[YOLOv8].csv`\n",
        "- `bboxes[DETR].csv`\n",
        "\n",
        "> `bboxes[V].csv`\n",
        ">\n",
        "> The single row in these files corresponds to a bounding box proposed by the `V` model, expressed in `xyxy` coordinates, complete with confidence level\n",
        "\n",
        "Following a fail-fast style, first the csvs are read in full, then the `torch.data.Dataset` classes are defined.\n",
        "\n",
        "For the training and the evaluation of the models presented in this report we have implemented four `torch.data.Dataset` classes. The overall meaning of these custom classes will be much clearer once read the following of the document.\n",
        "\n",
        "- `CocoDataset`\n",
        "- `Coco4MetricsDataset`\n",
        "- `Coco4TrainingDataset`\n",
        "- `Coco4ContrastiveDataset`\n",
        "\n",
        "> `CocoDataset`\n",
        ">\n",
        "> The single item is a tuple with the original image in `torch.Tensor` format, the prompts, the bounding boxes proposed by the visual model, and the bounding box ground truth.\n",
        "> `CocoDataset` filters the bounding boxes proposed by the model by setting a lowerbound on:\n",
        ">\n",
        "> - The confidence level\n",
        "> - The width in pixels of the bounding box\n",
        "> - The height in pixels of the bounding box\n",
        "\n",
        "> `Coco4MetricsDataset`\n",
        ">\n",
        "> The single item is a pair such that: the bounding boxes proposed by the visual model and the bounding box ground truth\n",
        "\n",
        "> `Coco4TrainingDataset`\n",
        ">\n",
        "> The single item, resembles the one of the aforementioned `CocoDataset`, but includes also the crops of the original image.\n",
        ">\n",
        "> In `Coco4TrainingDataset` we filter the bounding boxes as proposed in the `CocoDataset`.\n",
        "> In addition, `Coco4TrainingDataset` filters items based on the number of bounding boxes found by the visual model.\n",
        "> In fact, during the training phase, the model can only learn if it has at least two options to choose from.\n",
        "\n",
        "> `Coco4ContrastiveDataset`\n",
        ">\n",
        "> The single item is a pair defined as follows: original image crop ground truth and the prompts.\n",
        "\n",
        "Since the `torch.data.Dataset` custom classes defined so far are characterized by items of variable size, it is necessary to define a custom collate function. In particular, in this report, we have definded the followings:\n",
        "\n",
        "- `unzip`\n",
        "- `augment`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LuZuAEl69bK6"
      },
      "source": [
        "#### 5.1 Dataset and type declaration"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mPp8Dv3F6WW8"
      },
      "outputs": [],
      "source": [
        "%%shell\n",
        "if ! [ -d refcocog ]; then\n",
        "  gdown 1i-LHWSRp2F6--yhAi4IG3DiiCHmgE4cw &&\n",
        "  tar -xf refcocog.tar &&\n",
        "  rm refcocog.tar\n",
        "fi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W0wkQ9l961Rh"
      },
      "outputs": [],
      "source": [
        "path_root: str = os.path.join(\"refcocog\", \"\")\n",
        "path_annotations: str = os.path.join(path_root, \"annotations\", \"\")\n",
        "path_bboxes: str = os.path.join(path_root, \"bboxes\", \"\")\n",
        "path_images: str = os.path.join(path_root, \"images\", \"\")\n",
        "\n",
        "path_refs: str = os.path.join(path_annotations, \"refs.csv\")\n",
        "path_sentences: str = os.path.join(path_annotations, \"sentences.csv\")\n",
        "\n",
        "path_DETR: str = os.path.join(path_bboxes, \"bboxes[DETR].csv\")\n",
        "path_YOLOv5: str = os.path.join(path_bboxes, \"bboxes[YOLOv5].csv\")\n",
        "path_YOLOv8: str = os.path.join(path_bboxes, \"bboxes[YOLOv8].csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cSdOQiLh64XW"
      },
      "outputs": [],
      "source": [
        "Split = t.Literal[\"train\", \"test\", \"val\"]\n",
        "\n",
        "\n",
        "@dataclass\n",
        "class Ref:\n",
        "    ref_id: int  # unique id for refering expression\n",
        "    file_name: str  # file name of image relative to img_root\n",
        "    split: Split\n",
        "    xmin: float\n",
        "    ymin: float\n",
        "    xmax: float\n",
        "    ymax: float\n",
        "\n",
        "\n",
        "with open(path_refs, \"r\") as f:\n",
        "    raw = csv.DictReader(f)\n",
        "    refs: list[Ref] = [Ref(**row) for row in raw]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O7A7ACplxGRT"
      },
      "outputs": [],
      "source": [
        "@dataclass\n",
        "class Sentence:\n",
        "    ref_id: int  # unique id for refering expression\n",
        "    sent: str\n",
        "\n",
        "\n",
        "with open(path_sentences, \"r\") as f:\n",
        "    raw = csv.DictReader(f)\n",
        "    sentences: list[Sentence] = [Sentence(**row) for row in raw]\n",
        "\n",
        "\n",
        "id2sents: dict[int, list[str]] = groupby(\n",
        "    sentences, lambda x: x.ref_id, lambda x: x.sent\n",
        ")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JgmKrn_JxMaQ"
      },
      "outputs": [],
      "source": [
        "@dataclass\n",
        "class BBox:\n",
        "    file_name: str  # file name of image relative to img_root\n",
        "    xmin: float\n",
        "    ymin: float\n",
        "    xmax: float\n",
        "    ymax: float\n",
        "    confidence: float\n",
        "\n",
        "\n",
        "with open(path_DETR, \"r\") as f:\n",
        "    raw = csv.DictReader(f)\n",
        "    bboxes: list[BBox] = [BBox(**row) for row in raw]\n",
        "\n",
        "img2detr: dict[str, list[BBox]] = defaultdict(\n",
        "    list, groupby(bboxes, lambda x: x.file_name)\n",
        ")\n",
        "\n",
        "\n",
        "with open(path_YOLOv5, \"r\") as f:\n",
        "    raw = csv.DictReader(f)\n",
        "    bboxes: list[BBox] = [BBox(**row) for row in raw]\n",
        "\n",
        "img2yolov5: dict[str, list[BBox]] = defaultdict(\n",
        "    list, groupby(bboxes, lambda x: x.file_name)\n",
        ")\n",
        "\n",
        "\n",
        "with open(path_YOLOv8, \"r\") as f:\n",
        "    raw = csv.DictReader(f)\n",
        "    bboxes: list[BBox] = [BBox(**row) for row in raw]\n",
        "\n",
        "img2yolov8: dict[str, list[BBox]] = defaultdict(\n",
        "    list, groupby(bboxes, lambda x: x.file_name)\n",
        ")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZuYrGjZaxR0x"
      },
      "outputs": [],
      "source": [
        "TensorImage = UInt[torch.Tensor, \"3 H W\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CHbNwJ72gGoW"
      },
      "outputs": [],
      "source": [
        "class CocoDataset(Dataset[tuple[TensorImage, list[str], Float[torch.Tensor, \"X 4\"], Float[torch.Tensor, \"4\"]]]):\n",
        "    def __init__(\n",
        "        self,\n",
        "        split: Split,\n",
        "        img2bboxes: dict[str, list[BBox]],\n",
        "        limit: int = -1,\n",
        "    ):\n",
        "        self.items: list[\n",
        "            tuple[\n",
        "                str, list[str], Float[torch.Tensor, \"X 5\"], Float[torch.Tensor, \"1 4\"]\n",
        "            ]\n",
        "        ] = [\n",
        "            (img, sents, xyxys, xyxy)\n",
        "            for ref in refs\n",
        "            if ref.split == split\n",
        "            for img in [os.path.join(path_images, ref.file_name)]\n",
        "            for sents in [id2sents[ref.ref_id]]\n",
        "            for bboxes in [img2bboxes[ref.file_name]]\n",
        "            for xyxys in [\n",
        "                torch.tensor([\n",
        "                    (bbox.xmin, bbox.ymin, bbox.xmax, bbox.ymax)\n",
        "                    for bbox in bboxes\n",
        "                    if bbox.confidence > .25  # lower bound on confidence\n",
        "                ])\n",
        "            ]\n",
        "            for xyxy in [torch.tensor([(ref.xmin, ref.ymin, ref.xmax, ref.ymax)])]\n",
        "        ]\n",
        "        self.len: int = len(self.items) if limit < 0 else min(limit, len(self.items))\n",
        "\n",
        "    def __len__(self) -> int:\n",
        "        return self.len\n",
        "\n",
        "    def __getitem__(\n",
        "        self, index: int\n",
        "    ) -> tuple[\n",
        "        TensorImage, list[str], Float[torch.Tensor, \"X 5\"], Float[torch.Tensor, \"1 4\"]\n",
        "    ]:\n",
        "        file_name, sents, xyxys, xyxy = self.items[index]\n",
        "        return read_image(file_name, ImageReadMode.RGB).to(device), sents, xyxys, xyxy\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "class Coco4MetricsDataset(Dataset[tuple[Float[torch.Tensor, 'X 5'], Float[torch.Tensor, '1 4']]]):\n",
        "\n",
        "    def __init__(\n",
        "        self,\n",
        "        split: Split,\n",
        "        img2bboxes: dict[str, list[BBox]],\n",
        "        limit: int = -1,\n",
        "    ):\n",
        "        self.items: list[tuple[Float[torch.Tensor, 'X 5'], Float[torch.Tensor, '1 4']]] = [\n",
        "            (xyxys, xyxy)\n",
        "            for ref in refs\n",
        "            if ref.split == split\n",
        "            for bboxes in [img2bboxes[ref.file_name]]\n",
        "            for xyxys in [torch.tensor([ (bbox.xmin, bbox.ymin, bbox.xmax, bbox.ymax, bbox.confidence) for bbox in bboxes ], dtype=torch.float)]\n",
        "            for xyxy in [torch.tensor([(ref.xmin, ref.ymin, ref.xmax, ref.ymax)], dtype=torch.float)]\n",
        "        ]\n",
        "        self.len: int = len(self.items) if limit < 0 else min(limit, len(self.items))\n",
        "\n",
        "    def __len__(self) -> int:\n",
        "        return self.len\n",
        "\n",
        "    def __getitem__(self, index: int) -> tuple[Float[torch.Tensor, 'X 5'], Float[torch.Tensor, '1 4']]:\n",
        "        return self.items[index]\n",
        "\n"
      ],
      "metadata": {
        "id": "ZrZzc81HdIrV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-JE23eygJRUl"
      },
      "outputs": [],
      "source": [
        "class Coco4TrainingDataset(\n",
        "    Dataset[\n",
        "        tuple[\n",
        "            list[TensorImage],\n",
        "            list[str],\n",
        "            int,\n",
        "            Float[torch.Tensor, \"crops 4\"],\n",
        "            Float[torch.Tensor, \"1 4\"],\n",
        "        ]\n",
        "    ]\n",
        "):\n",
        "    def __init__(\n",
        "        self,\n",
        "        split: Split,\n",
        "        img2bboxes: dict[str, list[BBox]],\n",
        "        limit: int = -1,\n",
        "    ):\n",
        "        self.items: list[\n",
        "            tuple[\n",
        "                str,\n",
        "                list[str],\n",
        "                int,\n",
        "                Float[torch.Tensor, \"X 4\"],\n",
        "                Float[torch.Tensor, \"1 4\"],\n",
        "            ]\n",
        "        ] = [\n",
        "            (img, sents, i, xyxys, xyxy)\n",
        "            for ref in refs\n",
        "            if ref.split == split\n",
        "            for img in [os.path.join(path_images, ref.file_name)]\n",
        "            for sents in [id2sents[ref.ref_id]]\n",
        "            for bboxes in [img2bboxes[ref.file_name]]\n",
        "            for xyxys in [\n",
        "                torch.tensor([\n",
        "                    (bbox.xmin, bbox.ymin, bbox.xmax, bbox.ymax)\n",
        "                    for bbox in bboxes\n",
        "                    if bbox.confidence > .25  # lower bound on confidence\n",
        "                    if bbox.xmax - bbox.xmin > 16  # lower bound on width\n",
        "                    if bbox.ymax - bbox.ymin > 16  # lower bound on heigth\n",
        "                ])\n",
        "            ]\n",
        "            if xyxys.shape[0] > 1 # lower bound on bbox per image\n",
        "            for xyxy in [\n",
        "                torch.tensor([(ref.xmin, ref.ymin, ref.xmax, ref.ymax)])\n",
        "            ]\n",
        "            for ious in [box_iou(xyxys, xyxy)]\n",
        "            if torch.max(ious).item() > .5  # ensure at least .5 of maximum IoU\n",
        "            for i in [torch.argmax(ious).item()]\n",
        "        ]\n",
        "        self.len: int = len(self.items) if limit < 0 else min(limit, len(self.items))\n",
        "\n",
        "    def __len__(self) -> int:\n",
        "        return self.len\n",
        "\n",
        "    def __getitem__(\n",
        "        self, index: int\n",
        "    ) -> tuple[\n",
        "        list[TensorImage],\n",
        "        list[str],\n",
        "        int,\n",
        "        Float[torch.Tensor, \"crops 4\"],\n",
        "        Float[torch.Tensor, \"1 4\"],\n",
        "    ]:\n",
        "        file_name, sents, i, xyxys, xyxy = self.items[index]\n",
        "        img: TensorImage = read_image(file_name, ImageReadMode.RGB).to(device)\n",
        "\n",
        "        xywhs: Int[torch.Tensor, \"X 4\"] = box_convert(xyxys, in_fmt=\"xyxy\", out_fmt=\"xywh\").round().int()\n",
        "\n",
        "        crops: list[TensorImage] = [\n",
        "            crop(img, top=y, left=x, height=h, width=w)\n",
        "            for xywh in xywhs\n",
        "            for [x, y, w, h] in [xywh.tolist()]\n",
        "        ]\n",
        "\n",
        "        return crops, sents, i, xyxys, xyxy\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EzFS_QvJdccS"
      },
      "outputs": [],
      "source": [
        "class Coco4ContrastiveDataset(Dataset[tuple[TensorImage, list[str]]]):\n",
        "    def __init__(\n",
        "        self,\n",
        "        split: Split,\n",
        "        limit: int = -1,\n",
        "    ):\n",
        "        self.items: list[tuple[str, list[str], Float[torch.Tensor, \"1 4\"]]] = [\n",
        "            (img, sents, xyxy)\n",
        "            for ref in refs\n",
        "            if ref.split == split\n",
        "            for img in [os.path.join(path_images, ref.file_name)]\n",
        "            for sents in [id2sents[ref.ref_id]]\n",
        "            for xyxy in [torch.tensor([(ref.xmin, ref.ymin, ref.xmax, ref.ymax)])]\n",
        "        ]\n",
        "        self.len: int = len(self.items) if limit < 0 else min(limit, len(self.items))\n",
        "\n",
        "    def __len__(self) -> int:\n",
        "        return self.len\n",
        "\n",
        "    def __getitem__(self, index: int) -> tuple[TensorImage, list[str]]:\n",
        "        file_name, sents, xyxy = self.items[index]\n",
        "        img: TensorImage = read_image(file_name, ImageReadMode.RGB).to(device)\n",
        "\n",
        "        xywh: Int[torch.Tensor, \"1 4\"] = box_convert(xyxy, in_fmt=\"xyxy\", out_fmt=\"xywh\").round().int()\n",
        "        [[x, y, w, h]] = xywh.tolist()\n",
        "\n",
        "        return crop(img, top=y, left=x, height=h, width=w), sents\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eH9aI-TKooEi"
      },
      "source": [
        "## 6 Evaluation metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MtXUjxuNymH9"
      },
      "source": [
        "Recent works [12][13] have shown that even subtle changes in the finetuning process can lead to surprisingly large differences in the final performance. In order to provide valuable comparisons between our solution proposals and to ensure that the implemented models are performing as intended, it is essential to evaluate them according to appropriate metrics. With the aim of quantitatively measure the capabilities of our algorithms, we have considered the following criteria."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fh2X1lulzFYU"
      },
      "source": [
        "### 6.1 Localization accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qIgSYWRFzR8k"
      },
      "source": [
        "Localization accuracy (Formula 1) measures how accurately the fine-tuned network can ground the localized object to a language description. Intersection over Union (IoU) is a common measure of localization accuracy (Figure 6). In particular, we keep track of:\n",
        "*   mean intersection over union (**mIoU**)\n",
        "*   the fraction of correct predictions with respect to the total number of processed samples, considering an IoU threshold of 0.3 (**mAP [IoU .3]**). That is, the predicted bounding box $\\hat{b}$ is considered correct (true positive) if the IoU between $\\hat{b}$ and the ground-truth bounding box $b$ is at least 30%. This metric is usually referred to by the literature as *mean Average Precision* (mAP) [15].\n",
        "* the fraction of correct predictions with respect to the total number of processed samples, considering an IoU threshold of 0.5 (**mAP [IoU .5]**)\n",
        "* the fraction of correct predictions with respect to the total number of processed samples, considering an IoU threshold of 0.7 (**mAP [IoU .7]**)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4i2O8XOCzJi9"
      },
      "source": [
        "### 6.2 Semantic similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nNk4COYlzHrT"
      },
      "source": [
        "Semantic similarity (Formula 2) measures the similarity between the predicted bounding box $\\hat{b}$ and the portion of the image which corresponds to the ground-truth bounding box $b$. Consider the output proposed in Figure 7. The neural network predicts the red bounding box $\\hat{b}$ while in the dataset the green bounding box $b$ is the annotated ground-truth. Evidently, the algorithm response does not represent a true mistake. Conceptually, the two portions of the picture delimited by $\\hat{b}$ and $b$ are semantically equivalent. For the purpose of computing the semantic similarity between two crops we evaluate the mean cosine similarity (**mCos**) and the mean euclidean distance (**mED**) between the $\\hat{b}_z$s and the $b_z$s which are the latent space representations of the predicted bounding boxes $\\hat{b}_z$s and the ground-truth bounding boxes $b$s respectively. In order to achieve a fair comparison, we consistently use the original CLIP visual encoder (`encode_image()` method) to geometrically represent a cropped portion of the image with a vector of size 1024 (Figure 8)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znATC6cYyn93"
      },
      "source": [
        "**Formula 1**\n",
        "\n",
        "$d(b_z, \\hat{b}_z) = ||b_z - \\hat{b}||$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ggWNECXfyqb_"
      },
      "source": [
        "**Formula 2**\n",
        "\n",
        "$S_C(b_z, \\hat{b}_z) = \\frac{b_z \\cdot \\hat{b}_z}{\\lVert{} b_z \\rVert{} \\lVert{}\\hat{b}_z\\rVert{}} = \\frac{\\sum_{i=1}^{n}b_{z_i}\\hat{b}_{z_i}}{\\sqrt{\\sum_{i=1}^{n}b_{z_i}^2 \\cdot \\sum_{i=1}^{n}\\hat{b}_{z_i}^2}}$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-fidA2eytqR"
      },
      "source": [
        "> ![iou.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/06.png)\n",
        "> \n",
        "> **Figure 6**\n",
        "> \n",
        "> Intersection over Union\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-OO54Mnpyv-y"
      },
      "source": [
        "> ![semantic1.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/07.png)\n",
        "> \n",
        "> **Figure 7**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bbtxwxv7yyXC"
      },
      "source": [
        "> ![semantic2.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/08.png)\n",
        "> \n",
        "> **Figure 8**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 6.3 Code"
      ],
      "metadata": {
        "id": "GHdhRwJ3eExD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def eval_step(\n",
        "    model: nn.Module,\n",
        "    data_loader: DataLoader[tuple[TensorImage, list[str], Float[torch.Tensor, \"X 4\"], Float[torch.Tensor, \"4\"]]],\n",
        "    img_preprocess: t.Callable[[TensorImage], Float[torch.Tensor, \"3 244 244\"]],\n",
        ") -> pd.DataFrame:\n",
        "    model.eval()\n",
        "\n",
        "    ious: list[float] = []\n",
        "    coss: list[float] = []\n",
        "    euds: list[float] = []\n",
        "\n",
        "    with torch.inference_mode():\n",
        "        img: TensorImage\n",
        "        prompts: list[str]\n",
        "        xyxys: Float[torch.Tensor, \"crops 4\"]\n",
        "        xyxy: Float[torch.Tensor, \"4\"]\n",
        "\n",
        "        progress = tqdm(data_loader, desc=\"eval\")\n",
        "\n",
        "        for iter, (img, prompts, xyxys, true_xyxy) in enumerate(progress):\n",
        "\n",
        "            if xyxys.shape[0] == 0:\n",
        "                xyxys = torch.tensor((0, 0, img.shape[3], img.shape[2]))\n",
        "\n",
        "            # from xyxys to crops\n",
        "            xywhs: Int[torch.Tensor, \"X 4\"] = box_convert(xyxys, in_fmt=\"xyxy\", out_fmt=\"xywh\").round().int()\n",
        "\n",
        "            crops: list[TensorImage] = [\n",
        "                crop(img, top=y, left=x, height=h, width=w)\n",
        "                for xywh in xywhs\n",
        "                for [x, y, w, h] in [xywh.tolist()]\n",
        "            ]\n",
        "\n",
        "            # from true_xyxy to true_crop\n",
        "            true_xywh: Int[torch.Tensor, \"1 4\"] = box_convert(true_xyxy, in_fmt=\"xyxy\", out_fmt=\"xywh\").round().int()\n",
        "\n",
        "            true_crop: TensorImage\n",
        "            [true_crop] = [\n",
        "                crop(img, top=y, left=x, height=h, width=w)\n",
        "                for xywh in true_xywh\n",
        "                for [x, y, w, h] in [xywh.tolist()]\n",
        "            ]\n",
        "\n",
        "            # forward pass\n",
        "            model_output: Float[torch.Tensor, \"crops\"] = model(crops, prompts)\n",
        "\n",
        "            # get index of the predicted bounding box to compute IoU accuracy\n",
        "            pred_i: int = torch.argmax(model_output).item()\n",
        "\n",
        "            # get predicted bounding\n",
        "            pred_xyxy: Float[torch.Tensor, \"1 4\"] = xyxys[pred_i].unsqueeze(0)\n",
        "\n",
        "            iou: float = box_iou(true_xyxy, pred_xyxy).item()\n",
        "            ious.append(iou)\n",
        "\n",
        "            true_z: Float[torch.Tensor, \"1 1024\"] = clip_frozen_img_encoder(img_preprocess(true_crop).unsqueeze(0))\n",
        "            pred_z: Float[torch.Tensor, \"1 1024\"] = clip_frozen_img_encoder(img_preprocess(crops[pred_i]).unsqueeze(0))\n",
        "\n",
        "            cos: float = torch.nn.functional.cosine_similarity(true_z, pred_z).item()\n",
        "            coss.append(cos)\n",
        "\n",
        "            eud: float = torch.cdist(true_z, pred_z, p=2).item()\n",
        "            euds.append(eud)\n",
        "\n",
        "        return pd.DataFrame(\n",
        "            {\n",
        "                \"iou\": ious,\n",
        "                \"cos similarity\": coss,\n",
        "                \"euclidean distance\": euds,\n",
        "            }\n",
        "        )\n",
        "\n"
      ],
      "metadata": {
        "id": "a_LAz6SeeF5s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def showtime(\n",
        "    model: nn.Module,\n",
        "    data_loader: DataLoader[tuple[TensorImage, list[str], Float[torch.Tensor, \"X 4\"], Float[torch.Tensor, \"4\"]]],\n",
        "    writer: SummaryWriter,\n",
        "    global_step: int,\n",
        ") -> None:\n",
        "    model.eval()\n",
        "\n",
        "    with torch.inference_mode():\n",
        "        img: TensorImage\n",
        "        prompts: list[str]\n",
        "        xyxys: Float[torch.Tensor, \"crops 4\"]\n",
        "        xyxy: Float[torch.Tensor, \"4\"]\n",
        "\n",
        "        progress = tqdm(data_loader, desc=\"showtime\")\n",
        "\n",
        "        for iter, (img, prompts, xyxys, true_xyxy) in zip(it.count(1), progress):\n",
        "            true_i: int = best_bbox(xyxys, true_xyxy)\n",
        "\n",
        "            # from xyxys to crops\n",
        "            xywhs: Int[torch.Tensor, \"X 4\"] = box_convert(xyxys, in_fmt=\"xyxy\", out_fmt=\"xywh\").round().int()\n",
        "\n",
        "            crops: list[TensorImage] = [\n",
        "                crop(img, top=y, left=x, height=h, width=w)\n",
        "                for xywh in xywhs\n",
        "                for [x, y, w, h] in [xywh.tolist()]\n",
        "            ]\n",
        "\n",
        "            # forward pass\n",
        "            model_output: Float[torch.Tensor, \"crops\"] = model(crops, prompts)\n",
        "\n",
        "            # get index of the predicted bounding box to compute IoU accuracy\n",
        "            pred_i: int = torch.argmax(model_output).item()\n",
        "\n",
        "            # https://github.com/pytorch/pytorch/issues/65449\n",
        "            writer.add_image_with_boxes(\n",
        "                tag=f\"{iter}: {' \u00b6 '.join(prompts)}\",\n",
        "                img_tensor=img,\n",
        "                box_tensor=torch.stack((xyxys[pred_i], xyxys[true_i], true_xyxy.squeeze())),\n",
        "                labels=[\"prediction\", \"best region proposal\", \"ground truth\"],\n",
        "                global_step=global_step,\n",
        "            )\n",
        "\n"
      ],
      "metadata": {
        "id": "7o46_5KceIij"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def compare(reports: dict[str, pd.DataFrame]) -> pd.DataFrame:\n",
        "    return pd.DataFrame(\n",
        "        {\n",
        "            \"mAP[IoU .3]\": [(report[\"iou\"] >= 0.3).sum() / report[\"iou\"].count() for report in reports.values()],\n",
        "            \"mAP[IoU .5]\": [(report[\"iou\"] >= 0.5).sum() / report[\"iou\"].count() for report in reports.values()],\n",
        "            \"mAP[IoU .7]\": [(report[\"iou\"] >= 0.7).sum() / report[\"iou\"].count() for report in reports.values()],\n",
        "            \"mIoU\": [report[\"iou\"].mean() for report in reports.values()],\n",
        "            \"mCos\": [report[\"cos similarity\"].mean() for report in reports.values()],\n",
        "            \"mED\": [report[\"euclidean distance\"].mean() for report in reports.values()],\n",
        "        },\n",
        "        index=reports.keys(),\n",
        "    )"
      ],
      "metadata": {
        "id": "TyL-1bIVePEr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pJDSwl6Zp1-b"
      },
      "source": [
        "## 7 Baseline\n",
        "\n",
        "`BASELINE`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vjd2T6W6wnl4"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir ./assets/baseline/runs --port 6001"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qAePwXW3ofoS"
      },
      "source": [
        "At the beginning of our project we have implemented a training free baseline algorithm. The development of this solution, hereafter usually referred to as `BASELINE`, has been profitable since it allowed us to familiarize ourselves with the task of visual grounding, the dataset being used and the CLIP model. Moreover the evaluation of the obtained results has pointed out several interesting aspects including things that can be improved and an approximate understanding of the performance that our further solutions should obtain.\n",
        "\n",
        "The purpose of this section is to describe how the baseline algorithm works, report the obtained performance and provide a readable implementation of the described functionalities.\n",
        "\n",
        "The method is a training-free approach that combines CLIP zero-shot with a Yolo architecture [14]. More in depth, we rely on the Yolov5 implementation provided by TorchHub at the following link [Ultralytics Yolov5](https://pytorch.org/hub/ultralytics_yolov5/). The computational process involves extracting all the bounding boxes proposed by Yolo and evaluating their similarity with the textual query. In order to make comparisons between crops and prompts we rely on CLIP visual encoder (`encode_image()`) and CLIP text encoder (`encode_text()`) respectively, in order to map texts and images into the same latent space. This done, visual and textual prompts are represented as vectors belonging to the same 1024 dimensional vector space. At this point, the vectors which correspond to the embedded crops ($\\hat{b}_{z_1}, ..., \\hat{b}_{z_N}$) can be compared with the prompt encoding ($p$) using cosine similarity function ($S_C$). At the end of the day, the output of the execution is the bounding box corresponding to the vector $\\hat{b}_{z_i} \\in \\{\\hat{b}_{z_1}, ..., \\hat{b}_{z_N}\\}$ characterized by the highest similarity with the latent space representation of the textual prompt. For the sake of clarity we propose a schematic illustration of the overall architecture in Figure 9 and Figure 10.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xIalT5Ma1EDU"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/09.png)\n",
        "> \n",
        "> **Figure 9**\n",
        "> \n",
        "> The first step of the baseline is to perform object detection with Yolov5 algorithm."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fzR90cy-0fLs"
      },
      "source": [
        "> ![baseline.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/10.png)\n",
        "> \n",
        "> **Figure 10**\n",
        "> \n",
        "> The purpose of the second step of the baseline algorithm is to compare the vector embeddings of the previously extracted crops with the latent space representation of the input prompt $p$. The overall output is the bounding box $\\hat{b}_{z_i}$ which corresponds to the vector characterized by the highest similarity with respect to vector $p$. Reasonably, we rely on the cosine similarity function $S_C$ to do the comparison."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q6lIAW2L2sBN"
      },
      "source": [
        "### 7.1 Results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxz2QkQh0ZoH"
      },
      "source": [
        "Although our zero-shot `BASELINE` involves no additional training at all, it achieves very good performance on our downstream task. The obtained results are reported in the following table."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4rgihq6J2t1A"
      },
      "source": [
        "### 7.2 Observations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3o0pkxE1qP8"
      },
      "source": [
        "In the described two steps `BASELINE` algorithm the comparison of the image crops embeddings and the latent representation of the given textual prompt in order to identify the bounding box which is most similar to the given description, is computed among the regions of the input image proposed by Yolov5. As a consequence, the goodness of the predicted bounding box at the end of the execution is inevitably strongly related to the quality of the bounding boxes proposed by the former detection step. More broadly, being $\\hat{b}_{\\text{max}}$ the bounding box predicted by Yolov5 characterized by the highest intersection over union with the dataset ground truth bounding box, the localization accuracy of the `BASELINE` have an upper bound which is the average intersection over union between the $\\hat{b}_{\\text{max}}$s and the corresponding ground truths. In this regard, a good starting point to improve the performance is to replace Yolov5 with another region proposal algorithm which succeeds in suggesting better bounding boxes, i.e. rectangles with a higher intersection over union with the annotated ground truth.\n",
        "\n",
        "By means of a manual inspection of the regions proposed by Yolo, we have noticed that some crops are very small areas characterized by few pixels (Figure 11). In general, it does not make sense to consider these portions which clearly make the overall computation more expensive. A simple solution to overcome this is to discard all the bounding boxes whose edges are both smaller than a given threshold.\n",
        "\n",
        "We assess that the two steps pipeline of the `BASELINE` algorithm in which the predicted bounding box is chosen on top of the regions proposed in the first stage of the algorithm, is reasonably appropriate to tackle the problem. In the process, CLIP is used to map the image crops and the input prompt into a mutual latent space. The final answer of the algorithm strongly depends on the ability of CLIP to extract good features. For the purpose of improving the performance of our implemented solution, we believe that the most promising strategy is to apply transfer learning to fine-tune the CLIP image encoder and text encoder architectures in order to extract features which are more discriminative for our narrow domain. Intuitively, from a high level point of view, using more specialized features for the task at hand, leads necessarily to a better and more valuable end-to-end representation of visual and textual prompts, and therefore a finest comprehension of the scene. In Section 10 we present the solutions that we have come up with in order to further improve performance via supervised finetuning.\n",
        "\n",
        "In our reference dataset there are a lot of samples which include spatial relationships (Figure 12). In general, the baseline model has displayed good results on appearance-based descriptions that are independent of viewer perspective. However, the algorithm struggles in the presence of location words such as \"the dog on the left\", in the textual descriptions. We consider this limitation and describe our attempt to mitigate it in Section 12.\n",
        "\n",
        "Finally, in this first implementation, we always based our predictions on a single textual prompt which describes the portion of interest. However, in the dataset there are sometimes  more than one description for a given ground truth bounding box. In the following of this paper we describe our approach to consider more than one prompt in order to output more accurate predictions. More widely, in this regard in Section 13, we further investigate the chance of using data augmentation to improve the generalization capabilities of our final model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t7REqYcgUFcN"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/11.png)\n",
        "> \n",
        "> **Figure 11**\n",
        "> \n",
        "> Three crops proposed by Yolo. Evidently, two of them are meaningless and are too small to accomodate interesting portions of the picture."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ov3oyuFPTazV"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/12.png)\n",
        "> \n",
        "> **Figure 12**\n",
        "> \n",
        "> We have notices that the `BASELINE` algorithm has severe issues in dealing with spatial relationships. For instance in this case it has difficulties in choosing between the two dogs in the picture."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.3 Code"
      ],
      "metadata": {
        "id": "JN1H76lvcWMn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clip_model, clip_preprocessor = clip.load(\"RN50\", device=device)\n",
        "clip_model.float()\n",
        "clip_model.eval()\n",
        "\n",
        "for p in clip_model.parameters():\n",
        "    p.requires_grad = False"
      ],
      "metadata": {
        "id": "ImpXZp1ucbVn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def transform(n_px: int) -> Compose:\n",
        "    \"\"\"\n",
        "    https://github.com/openai/CLIP/blob/a1d071733d7111c9c014f024669f959182114e33/clip/clip.py#L75-L86\n",
        "    \"\"\"\n",
        "    return Compose([\n",
        "        ConvertImageDtype(torch.float),\n",
        "        Resize(n_px, interpolation=InterpolationMode.BICUBIC, antialias=True),\n",
        "        CenterCrop(n_px),\n",
        "        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),\n",
        "    ])\n",
        "\n",
        "\n",
        "preprocess: Compose = transform(224)"
      ],
      "metadata": {
        "id": "He7ku3Yccohf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipFrozenImgEnc(nn.Module):\n",
        "    def forward(\n",
        "        self, image: Float[torch.Tensor, \"crops 3 244 244\"]\n",
        "    ) -> Float[torch.Tensor, \"crops 1024\"]:\n",
        "        with torch.no_grad():\n",
        "            return clip_model.encode_image(image).float()\n",
        "\n",
        "\n",
        "class ClipFrozenTxtEnc(nn.Module):\n",
        "    def forward(\n",
        "        self, text: Int[torch.Tensor, \"prompts 77\"]\n",
        "    ) -> Float[torch.Tensor, \"prompts 1024\"]:\n",
        "        with torch.no_grad():\n",
        "            return clip_model.encode_text(text).float()"
      ],
      "metadata": {
        "id": "nFbMQ_mjcp5n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "clip_frozen_img_encoder: ClipFrozenImgEnc = ClipFrozenImgEnc()\n",
        "clip_frozen_txt_encoder: ClipFrozenTxtEnc = ClipFrozenTxtEnc()"
      ],
      "metadata": {
        "id": "nuzByqQrcsqJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "P_itJlG2eVGW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipWrapper(nn.Module):\n",
        "    def __init__(self, clip_model: CLIP):\n",
        "        super().__init__()\n",
        "        self.img_preprocess: Compose = preprocess\n",
        "        self.txt_preprocess: t.Callable[[t.Union[str, list[str]]], Float[torch.Tensor, \"77\"]] = clip.tokenize\n",
        "        self.clip_model: CLIP = clip_model\n",
        "\n",
        "    def forward(\n",
        "        self, crops: list[TensorImage], prompts: list[str]\n",
        "    ) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        with torch.no_grad():\n",
        "            # step 1: preprocess crops as required by the visual encoder\n",
        "            crops_preprocessed: Float[torch.Tensor, \"crops 3 244 244\"] = torch.stack([\n",
        "                self.img_preprocess(crop)\n",
        "                for crop in crops\n",
        "            ])\n",
        "\n",
        "            # step 2: preprocess prompts as required by the text encoder\n",
        "            prompts_preprocessed: Int[torch.Tensor, \"prompts 77\"] = self.txt_preprocess(prompts)\n",
        "\n",
        "            similarity_matrix: Float[torch.Tensor, \"prompts crops\"]\n",
        "            _, similarity_matrix = self.clip_model(\n",
        "                crops_preprocessed,\n",
        "                prompts_preprocessed,\n",
        "            )\n",
        "\n",
        "            return torch.mean(similarity_matrix, dim=0)"
      ],
      "metadata": {
        "id": "2IAd1iWBeWgY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_summary(\n",
        "    clip_model\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HF3Lt6cpeYYT",
        "outputId": "fc806288-a4a5-4ac2-fbbc-a67c62a1a633"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "======================================================================================================================================================\n",
              "Layer (type:depth-idx)                             Input Shape               Output Shape              Param #                   Trainable\n",
              "======================================================================================================================================================\n",
              "CLIP                                               [5, 3, 244, 244]          [5, 2]                    563,713                   False\n",
              "\u251c\u2500ModifiedResNet: 1-1                              [5, 3, 244, 244]          [5, 1024]                 --                        False\n",
              "\u2502    \u2514\u2500Conv2d: 2-1                                 [5, 3, 244, 244]          [5, 32, 122, 122]         (864)                     False\n",
              "\u2502    \u2514\u2500BatchNorm2d: 2-2                            [5, 32, 122, 122]         [5, 32, 122, 122]         (64)                      False\n",
              "\u2502    \u2514\u2500ReLU: 2-3                                   [5, 32, 122, 122]         [5, 32, 122, 122]         --                        --\n",
              "\u2502    \u2514\u2500Conv2d: 2-4                                 [5, 32, 122, 122]         [5, 32, 122, 122]         (9,216)                   False\n",
              "\u2502    \u2514\u2500BatchNorm2d: 2-5                            [5, 32, 122, 122]         [5, 32, 122, 122]         (64)                      False\n",
              "\u2502    \u2514\u2500ReLU: 2-6                                   [5, 32, 122, 122]         [5, 32, 122, 122]         --                        --\n",
              "\u2502    \u2514\u2500Conv2d: 2-7                                 [5, 32, 122, 122]         [5, 64, 122, 122]         (18,432)                  False\n",
              "\u2502    \u2514\u2500BatchNorm2d: 2-8                            [5, 64, 122, 122]         [5, 64, 122, 122]         (128)                     False\n",
              "\u2502    \u2514\u2500ReLU: 2-9                                   [5, 64, 122, 122]         [5, 64, 122, 122]         --                        --\n",
              "\u2502    \u2514\u2500AvgPool2d: 2-10                             [5, 64, 122, 122]         [5, 64, 61, 61]           --                        --\n",
              "\u2502    \u2514\u2500Sequential: 2-11                            [5, 64, 61, 61]           [5, 256, 61, 61]          --                        False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-1                        [5, 64, 61, 61]           [5, 256, 61, 61]          (75,008)                  False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-2                        [5, 256, 61, 61]          [5, 256, 61, 61]          (70,400)                  False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-3                        [5, 256, 61, 61]          [5, 256, 61, 61]          (70,400)                  False\n",
              "\u2502    \u2514\u2500Sequential: 2-12                            [5, 256, 61, 61]          [5, 512, 30, 30]          --                        False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-4                        [5, 256, 61, 61]          [5, 512, 30, 30]          (379,392)                 False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-5                        [5, 512, 30, 30]          [5, 512, 30, 30]          (280,064)                 False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-6                        [5, 512, 30, 30]          [5, 512, 30, 30]          (280,064)                 False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-7                        [5, 512, 30, 30]          [5, 512, 30, 30]          (280,064)                 False\n",
              "\u2502    \u2514\u2500Sequential: 2-13                            [5, 512, 30, 30]          [5, 1024, 15, 15]         --                        False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-8                        [5, 512, 30, 30]          [5, 1024, 15, 15]         (1,512,448)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-9                        [5, 1024, 15, 15]         [5, 1024, 15, 15]         (1,117,184)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-10                       [5, 1024, 15, 15]         [5, 1024, 15, 15]         (1,117,184)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-11                       [5, 1024, 15, 15]         [5, 1024, 15, 15]         (1,117,184)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-12                       [5, 1024, 15, 15]         [5, 1024, 15, 15]         (1,117,184)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-13                       [5, 1024, 15, 15]         [5, 1024, 15, 15]         (1,117,184)               False\n",
              "\u2502    \u2514\u2500Sequential: 2-14                            [5, 1024, 15, 15]         [5, 2048, 7, 7]           --                        False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-14                       [5, 1024, 15, 15]         [5, 2048, 7, 7]           (6,039,552)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-15                       [5, 2048, 7, 7]           [5, 2048, 7, 7]           (4,462,592)               False\n",
              "\u2502    \u2502    \u2514\u2500Bottleneck: 3-16                       [5, 2048, 7, 7]           [5, 2048, 7, 7]           (4,462,592)               False\n",
              "\u2502    \u2514\u2500AttentionPool2d: 2-15                       [5, 2048, 7, 7]           [5, 1024]                 14,789,632                False\n",
              "\u251c\u2500Embedding: 1-2                                   [2, 77]                   [2, 77, 512]              (25,296,896)              False\n",
              "\u251c\u2500Transformer: 1-3                                 [77, 2, 512]              [77, 2, 512]              --                        False\n",
              "\u2502    \u2514\u2500Sequential: 2-16                            [77, 2, 512]              [77, 2, 512]              --                        False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-17           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-18           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-19           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-20           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-21           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-22           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-23           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-24           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-25           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-26           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-27           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u2502    \u2502    \u2514\u2500ResidualAttentionBlock: 3-28           [77, 2, 512]              [77, 2, 512]              (3,152,384)               False\n",
              "\u251c\u2500LayerNorm: 1-4                                   [2, 77, 512]              [2, 77, 512]              (1,024)                   False\n",
              "======================================================================================================================================================\n",
              "Total params: 102,007,137\n",
              "Trainable params: 0\n",
              "Non-trainable params: 102,007,137\n",
              "Total mult-adds (G): 32.67\n",
              "======================================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 1212.16\n",
              "Params size (MB): 296.19\n",
              "Estimated Total Size (MB): 1511.91\n",
              "======================================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "splits: list[Split] = [\"train\", \"val\", \"test\"]\n",
        "\n",
        "pd.concat(\n",
        "    [\n",
        "        pd.read_csv(f\"assets/baseline/eval-baseline-{split}.csv\", index_col=0)\n",
        "        for split in splits\n",
        "    ],\n",
        "    axis=1,\n",
        "    keys=splits,\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        },
        "id": "RhRURtYTea_3",
        "outputId": "ceb1487b-41b1-49c6-d18a-e11638387973"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              train                                            val  \\\n",
              "                iou cos similarity euclidean distance          iou   \n",
              "count  42226.000000   42226.000000       42226.000000  2573.000000   \n",
              "mean       0.548180       0.885833           0.836584     0.545417   \n",
              "std        0.394632       0.119882           0.467774     0.395396   \n",
              "min        0.000000       0.238326           0.000000     0.000000   \n",
              "25%        0.094556       0.822790           0.444478     0.085112   \n",
              "50%        0.749950       0.941208           0.693325     0.729440   \n",
              "75%        0.920812       0.976068           1.177713     0.921546   \n",
              "max        0.999468       1.000000           2.965670     0.990354   \n",
              "\n",
              "                                                test                 \\\n",
              "      cos similarity euclidean distance          iou cos similarity   \n",
              "count    2573.000000        2573.000000  5023.000000    5023.000000   \n",
              "mean        0.884527           0.840914     0.547995       0.883890   \n",
              "std         0.120132           0.466306     0.394547       0.123470   \n",
              "min         0.386646           0.078198     0.000000       0.305500   \n",
              "25%         0.820903           0.449173     0.099722       0.819754   \n",
              "50%         0.938441           0.705266     0.748046       0.940718   \n",
              "75%         0.975418           1.182641     0.921519       0.976637   \n",
              "max         0.999322           2.501004     0.995819       1.000000   \n",
              "\n",
              "                          \n",
              "      euclidean distance  \n",
              "count        5023.000000  \n",
              "mean            0.840413  \n",
              "std             0.473257  \n",
              "min             0.000000  \n",
              "25%             0.444113  \n",
              "50%             0.691364  \n",
              "75%             1.185348  \n",
              "max             2.962296  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-36f9b9c3-6516-473b-8558-84f6ff776d6d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"3\" halign=\"left\">train</th>\n",
              "      <th colspan=\"3\" halign=\"left\">val</th>\n",
              "      <th colspan=\"3\" halign=\"left\">test</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.548180</td>\n",
              "      <td>0.885833</td>\n",
              "      <td>0.836584</td>\n",
              "      <td>0.545417</td>\n",
              "      <td>0.884527</td>\n",
              "      <td>0.840914</td>\n",
              "      <td>0.547995</td>\n",
              "      <td>0.883890</td>\n",
              "      <td>0.840413</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.394632</td>\n",
              "      <td>0.119882</td>\n",
              "      <td>0.467774</td>\n",
              "      <td>0.395396</td>\n",
              "      <td>0.120132</td>\n",
              "      <td>0.466306</td>\n",
              "      <td>0.394547</td>\n",
              "      <td>0.123470</td>\n",
              "      <td>0.473257</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.238326</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.386646</td>\n",
              "      <td>0.078198</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.305500</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.094556</td>\n",
              "      <td>0.822790</td>\n",
              "      <td>0.444478</td>\n",
              "      <td>0.085112</td>\n",
              "      <td>0.820903</td>\n",
              "      <td>0.449173</td>\n",
              "      <td>0.099722</td>\n",
              "      <td>0.819754</td>\n",
              "      <td>0.444113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.749950</td>\n",
              "      <td>0.941208</td>\n",
              "      <td>0.693325</td>\n",
              "      <td>0.729440</td>\n",
              "      <td>0.938441</td>\n",
              "      <td>0.705266</td>\n",
              "      <td>0.748046</td>\n",
              "      <td>0.940718</td>\n",
              "      <td>0.691364</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.920812</td>\n",
              "      <td>0.976068</td>\n",
              "      <td>1.177713</td>\n",
              "      <td>0.921546</td>\n",
              "      <td>0.975418</td>\n",
              "      <td>1.182641</td>\n",
              "      <td>0.921519</td>\n",
              "      <td>0.976637</td>\n",
              "      <td>1.185348</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.999468</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.965670</td>\n",
              "      <td>0.990354</td>\n",
              "      <td>0.999322</td>\n",
              "      <td>2.501004</td>\n",
              "      <td>0.995819</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.962296</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-36f9b9c3-6516-473b-8558-84f6ff776d6d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-36f9b9c3-6516-473b-8558-84f6ff776d6d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-36f9b9c3-6516-473b-8558-84f6ff776d6d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-658592f7-413c-4832-bbab-24d265b358ce\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-658592f7-413c-4832-bbab-24d265b358ce')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-658592f7-413c-4832-bbab-24d265b358ce button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 1**"
      ],
      "metadata": {
        "id": "J3OTvbADPIvf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.read_csv(f\"assets/baseline/comparing.csv\", index_col=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "3iI7LdU_ecw8",
        "outputId": "d881ad5c-9809-4d2c-bb67-92d85a1cd7e2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       mAP[IoU .3]  mAP[IoU .5]  mAP[IoU .7]      mIoU      mCos       mED\n",
              "train     0.628239     0.571709     0.520414  0.548180  0.885833  0.836584\n",
              "val       0.628838     0.570152     0.514963  0.545417  0.884527  0.840914\n",
              "test      0.629305     0.568784     0.518813  0.547995  0.883890  0.840413"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1c779505-1272-45e9-a35c-ee9392986f13\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mAP[IoU .3]</th>\n",
              "      <th>mAP[IoU .5]</th>\n",
              "      <th>mAP[IoU .7]</th>\n",
              "      <th>mIoU</th>\n",
              "      <th>mCos</th>\n",
              "      <th>mED</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>train</th>\n",
              "      <td>0.628239</td>\n",
              "      <td>0.571709</td>\n",
              "      <td>0.520414</td>\n",
              "      <td>0.548180</td>\n",
              "      <td>0.885833</td>\n",
              "      <td>0.836584</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>0.628838</td>\n",
              "      <td>0.570152</td>\n",
              "      <td>0.514963</td>\n",
              "      <td>0.545417</td>\n",
              "      <td>0.884527</td>\n",
              "      <td>0.840914</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>test</th>\n",
              "      <td>0.629305</td>\n",
              "      <td>0.568784</td>\n",
              "      <td>0.518813</td>\n",
              "      <td>0.547995</td>\n",
              "      <td>0.883890</td>\n",
              "      <td>0.840413</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1c779505-1272-45e9-a35c-ee9392986f13')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1c779505-1272-45e9-a35c-ee9392986f13 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1c779505-1272-45e9-a35c-ee9392986f13');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-05eb415f-b83c-4c3a-a7fd-9150c85d4141\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-05eb415f-b83c-4c3a-a7fd-9150c85d4141')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-05eb415f-b83c-4c3a-a7fd-9150c85d4141 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 2**"
      ],
      "metadata": {
        "id": "u_mbSkRXPNCx"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BqDJn5AXbBAq"
      },
      "source": [
        "## 8 Approach"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YZA09W9Lpz2F"
      },
      "source": [
        "Motivated by these observations, in this work we propose a two steps joint embedding algorithm to solve the visual grounding problem. The general framework is depicted in Figure 13.\n",
        "\n",
        "The purpose of the first stage is to select a collection of regions of interest in the input image. To this end we need a deep learning model which predicts a set of bounding boxes which frequently contains a bounding box with high intersection over union with the ground truth bounding box of the RefCOCOg dataset.\n",
        "\n",
        "In the second phase we incorporate our fine-tuned versions of the CLIP visual and textual encoders to encode the image regions delimited by the proposed bounding boxes and the referring expression into the same vector space. By doing so we can locate the portion of the image which is more in line with the given natural language description by means of the cosine similarity function as proposed in our baseline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DlpPKG0Mp5cB"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/13.png)\n",
        "> \n",
        "> **Figure 13**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5RN8F5t-uNOe"
      },
      "source": [
        "## 9 Region proposal algorithms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAQWcGgZyz-g"
      },
      "source": [
        "In consonance with the schema proposed in Figure 13, the purpose of the first step of our algorithm is to propose potential regions of interest within the field of view. Understanding the location of the relevant objects in a given scene is a trivial task for humans. However, the development of a software agent with such a capability has been an uphill task until the turn of the last decade [15]. In recent years, the computer vision literature has proposed several models to accomplish this task motivated by a huge range of possible applications. In the `BASELINE` algorithm we have employed the widely used Yolov5 model. As mentioned above, the goodness of the bounding box proposed at the end of the overall execution is inherently related to the quality of the bounding boxes proposed by Yolov5. More in depth, being $\\hat{B}=\\{\\hat{b}_1, \u2026, \\hat{b}_N\\}$ the set of bounding boxes proposed by Yolov5 and $\\hat{b}_{\\text{max}} \\in \\hat{B}$ the proposed bounding box characterized by the highest intersection over union with the dataset ground truth bounding box $b$; the localization accuracy of the final predicted bounding box is necessarily limited superiorly by the IoU($\\hat{b}_{\\text{max}}, b$). In addition to this, also the cardinality of $B$ is not negligible. Actually, the computational complexity of the successive iterations depends on the number of bounding boxes to be evaluated. Based on this observations, we have tested multiple object detection algorithms made freely available by the authors with the aim of finding out the most performing one on our dataset. To this end, we have considered some of the state-of-the-art methods listed by paperswithcode.com [16]. Particularly, in this project we have executed Yolov5 [14], Yolov8 [17] and DETR [18]. For the sake of a proper comparison, all the algorithms have been configured with the same level of confidence. The obtained results are reported in Table 3. As we can see, though all the models perform decently, it turns out that DETR provides the best tradeoff between number of bounding boxes and average IoU with the dataset ground truth. Ultimately, in the following of the project all the presented results have been achieved on top of the regions proposed by DETR.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EamDGEc3COB7"
      },
      "source": [
        "### 9.1 Bounding box preprocessing\n",
        "As exhaustively written in Section 8 and depicted in Figure 13, once the first step of the overall algorithm has been completed, the obtained bounding boxes are not further refined in the following of the execution. Therefore, we have decided to preprocess our entire dataset in order to fill it with the bounding boxes proposed by the aforementioned object detectors. In doing so we have considerably speeded up the overall computational execution without incurring any loss of generality. A single epoch without this preprocessing last 90 minutes on average. On the other hand, with this enhancement we complete an epoch iteration in 50 minutes. As a consequence of this, we have been able to make more experiments and to train our models on more data for a longer time. We believe that this preprocessing can be potentially applied on countless deep learning domains. Hence, as an important contribution of our work we have made available at this GitHub repository the code to compute this preprocessing with whatever object detection algorithm. Moreover, in the repo, we have conveniently published `yolov5.csv`, `yolov8.csv` and `detr.csv` files including the results calculated by the three aforementioned object detectors."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 9.2 Code"
      ],
      "metadata": {
        "id": "DOFSpCAGddHs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def metrics(dataset: Dataset[tuple[Float[torch.Tensor, 'X 5'], Float[torch.Tensor, '1 4']]]) -> pd.DataFrame:\n",
        "\n",
        "    dataloader: DataLoader[tuple[Float[torch.Tensor, 'X 5'], Float[torch.Tensor, '1 4']]] = DataLoader(dataset, batch_size=None)\n",
        "    Z: Float[torch.Tensor, '1 5'] = torch.zeros(1, 5)\n",
        "\n",
        "    ious: list[float] = [ torch.max(box_iou(true_xyxy, torch.cat((Z, xyxys))[:, :4])).item() for xyxys, true_xyxy in tqdm(dataloader) ]\n",
        "    rs: list[int] = [ xyxys.shape[0] for xyxys, _ in tqdm(dataloader) ]\n",
        "\n",
        "    return pd.DataFrame({'iou': ious, '#': rs})"
      ],
      "metadata": {
        "id": "Yck5fpvLdex0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "splits: list[Split] = ['train', 'val', 'test']\n",
        "report: pd.DataFrame = pd.concat(\n",
        "    [\n",
        "        pd.concat(\n",
        "            [yolov5, yolov8, detr],\n",
        "            axis=1,\n",
        "            keys=['yolov5', 'yolov8', 'detr']\n",
        "        ).describe()\n",
        "        for split in splits\n",
        "        for yolov5 in [metrics(Coco4MetricsDataset(split, img2yolov5))]\n",
        "        for yolov8 in [metrics(Coco4MetricsDataset(split, img2yolov8))]\n",
        "        for detr in [metrics(Coco4MetricsDataset(split, img2detr))]\n",
        "    ],\n",
        "    axis=1,\n",
        "    keys=splits\n",
        ")"
      ],
      "metadata": {
        "id": "taGnY_zUdhHW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "report"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 383
        },
        "id": "cVL_MoCEdkLY",
        "outputId": "59d40ff8-887e-4cc6-b449-de3194ad84ab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              train                                                          \\\n",
              "             yolov5                      yolov8                        detr   \n",
              "                iou             #           iou             #           iou   \n",
              "count  42226.000000  42226.000000  42226.000000  42226.000000  42226.000000   \n",
              "mean       0.825308     11.371288      0.917952     11.666438      0.916711   \n",
              "std        0.183443      9.895845      0.114324      9.746315      0.082985   \n",
              "min        0.000000      0.000000      0.000000      1.000000      0.000000   \n",
              "25%        0.790363      5.000000      0.916166      5.000000      0.900589   \n",
              "50%        0.897469      8.000000      0.952232      9.000000      0.941296   \n",
              "75%        0.939805     15.000000      0.971201     15.000000      0.964255   \n",
              "max        0.999468    127.000000      0.998281    117.000000      0.998923   \n",
              "\n",
              "                             val                                         \\\n",
              "                          yolov5                    yolov8                \n",
              "                  #          iou            #          iou            #   \n",
              "count  42226.000000  2573.000000  2573.000000  2573.000000  2573.000000   \n",
              "mean      25.652181     0.823866    11.614069     0.914574    12.156627   \n",
              "std       23.022665     0.182910     9.821220     0.118846    10.222522   \n",
              "min        1.000000     0.000000     1.000000     0.026681     1.000000   \n",
              "25%        8.000000     0.778337     5.000000     0.912336     5.000000   \n",
              "50%       18.000000     0.899470     8.000000     0.951201     9.000000   \n",
              "75%       37.000000     0.939672    15.000000     0.970959    16.000000   \n",
              "max      100.000000     0.990354    72.000000     0.996324    87.000000   \n",
              "\n",
              "                                        test                            \\\n",
              "              detr                    yolov5                    yolov8   \n",
              "               iou            #          iou            #          iou   \n",
              "count  2573.000000  2573.000000  5023.000000  5023.000000  5023.000000   \n",
              "mean      0.915542    26.676642     0.825951    11.193510     0.916911   \n",
              "std       0.082728    23.541012     0.181452     9.891445     0.118269   \n",
              "min       0.103549     2.000000     0.000000     1.000000     0.000000   \n",
              "25%       0.899168     8.000000     0.784575     5.000000     0.916092   \n",
              "50%       0.940851    18.000000     0.896940     8.000000     0.953306   \n",
              "75%       0.963627    38.000000     0.940734    14.000000     0.971352   \n",
              "max       0.996266    99.000000     0.995819    96.000000     0.997200   \n",
              "\n",
              "                                              \n",
              "                           detr               \n",
              "                 #          iou            #  \n",
              "count  5023.000000  5023.000000  5023.000000  \n",
              "mean     11.474617     0.917533    25.201672  \n",
              "std       9.782079     0.080795    22.871364  \n",
              "min       1.000000     0.207010     1.000000  \n",
              "25%       5.000000     0.901468     8.000000  \n",
              "50%       8.000000     0.942469    17.000000  \n",
              "75%      14.000000     0.964712    35.000000  \n",
              "max      99.000000     0.998143   100.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-37767cf6-f72e-4613-b2e9-28e8422c55ed\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"6\" halign=\"left\">train</th>\n",
              "      <th colspan=\"6\" halign=\"left\">val</th>\n",
              "      <th colspan=\"6\" halign=\"left\">test</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"2\" halign=\"left\">yolov5</th>\n",
              "      <th colspan=\"2\" halign=\"left\">yolov8</th>\n",
              "      <th colspan=\"2\" halign=\"left\">detr</th>\n",
              "      <th colspan=\"2\" halign=\"left\">yolov5</th>\n",
              "      <th colspan=\"2\" halign=\"left\">yolov8</th>\n",
              "      <th colspan=\"2\" halign=\"left\">detr</th>\n",
              "      <th colspan=\"2\" halign=\"left\">yolov5</th>\n",
              "      <th colspan=\"2\" halign=\"left\">yolov8</th>\n",
              "      <th colspan=\"2\" halign=\"left\">detr</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "      <th>iou</th>\n",
              "      <th>#</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.825308</td>\n",
              "      <td>11.371288</td>\n",
              "      <td>0.917952</td>\n",
              "      <td>11.666438</td>\n",
              "      <td>0.916711</td>\n",
              "      <td>25.652181</td>\n",
              "      <td>0.823866</td>\n",
              "      <td>11.614069</td>\n",
              "      <td>0.914574</td>\n",
              "      <td>12.156627</td>\n",
              "      <td>0.915542</td>\n",
              "      <td>26.676642</td>\n",
              "      <td>0.825951</td>\n",
              "      <td>11.193510</td>\n",
              "      <td>0.916911</td>\n",
              "      <td>11.474617</td>\n",
              "      <td>0.917533</td>\n",
              "      <td>25.201672</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.183443</td>\n",
              "      <td>9.895845</td>\n",
              "      <td>0.114324</td>\n",
              "      <td>9.746315</td>\n",
              "      <td>0.082985</td>\n",
              "      <td>23.022665</td>\n",
              "      <td>0.182910</td>\n",
              "      <td>9.821220</td>\n",
              "      <td>0.118846</td>\n",
              "      <td>10.222522</td>\n",
              "      <td>0.082728</td>\n",
              "      <td>23.541012</td>\n",
              "      <td>0.181452</td>\n",
              "      <td>9.891445</td>\n",
              "      <td>0.118269</td>\n",
              "      <td>9.782079</td>\n",
              "      <td>0.080795</td>\n",
              "      <td>22.871364</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.026681</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.103549</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.207010</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.790363</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0.916166</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0.900589</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>0.778337</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0.912336</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0.899168</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>0.784575</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0.916092</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0.901468</td>\n",
              "      <td>8.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.897469</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>0.952232</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>0.941296</td>\n",
              "      <td>18.000000</td>\n",
              "      <td>0.899470</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>0.951201</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>0.940851</td>\n",
              "      <td>18.000000</td>\n",
              "      <td>0.896940</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>0.953306</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>0.942469</td>\n",
              "      <td>17.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.939805</td>\n",
              "      <td>15.000000</td>\n",
              "      <td>0.971201</td>\n",
              "      <td>15.000000</td>\n",
              "      <td>0.964255</td>\n",
              "      <td>37.000000</td>\n",
              "      <td>0.939672</td>\n",
              "      <td>15.000000</td>\n",
              "      <td>0.970959</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>0.963627</td>\n",
              "      <td>38.000000</td>\n",
              "      <td>0.940734</td>\n",
              "      <td>14.000000</td>\n",
              "      <td>0.971352</td>\n",
              "      <td>14.000000</td>\n",
              "      <td>0.964712</td>\n",
              "      <td>35.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.999468</td>\n",
              "      <td>127.000000</td>\n",
              "      <td>0.998281</td>\n",
              "      <td>117.000000</td>\n",
              "      <td>0.998923</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>0.990354</td>\n",
              "      <td>72.000000</td>\n",
              "      <td>0.996324</td>\n",
              "      <td>87.000000</td>\n",
              "      <td>0.996266</td>\n",
              "      <td>99.000000</td>\n",
              "      <td>0.995819</td>\n",
              "      <td>96.000000</td>\n",
              "      <td>0.997200</td>\n",
              "      <td>99.000000</td>\n",
              "      <td>0.998143</td>\n",
              "      <td>100.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-37767cf6-f72e-4613-b2e9-28e8422c55ed')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-37767cf6-f72e-4613-b2e9-28e8422c55ed button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-37767cf6-f72e-4613-b2e9-28e8422c55ed');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c2fa591b-5a45-4a9c-90b9-d854f5109ad5\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c2fa591b-5a45-4a9c-90b9-d854f5109ad5')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c2fa591b-5a45-4a9c-90b9-d854f5109ad5 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 3**"
      ],
      "metadata": {
        "id": "ZNxPEOrcPPnM"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5tbp8YEp3yO"
      },
      "source": [
        "## 10 Standard fine-tuning\n",
        "\n",
        "`net1` `net2` `net3` `net4`"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%tensorboard --logdir ./assets/standard-finetuning/runs --port 6002"
      ],
      "metadata": {
        "id": "G-ta-yLi8JaP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3D3zWSL96ZoJ"
      },
      "source": [
        "Given a set of proposed bounding boxes $\\hat{B}=\\{\\hat{b}_1, \u2026, \\hat{b}_N\\}$ and a natural language description $p$ referring to one of them, the purpose of the second step of our solution proposal, is to map both visual and textual prompts into a mutual vector space. In the latent space, the bounding boxes and the referring expression are represented as vectors of features. Given that these vectors share a common geometrical space, they have the same dimensionality. As a consequence, we can mathematically compare the visual information within the bounding boxes with the semantic of the provided description simply using cosine similarity. In this way, our algorithm can state which is the image region more in line with the input expression.\n",
        "\n",
        "Evidently, the tough part of this road map is the definition of a meaningful mapping between visual and textual information. To this end, we rely on the powerful pretrained CLIP model recently proposed by OpenAI [2]. More specifically, the deep neural network provides a visual encoder $\\Psi: I \\rightarrow Z$ and a textual encoder $\\Gamma: P \\rightarrow Z$ such that given respectively an image $i \\in I$ and a text $p \\in P$ as input, they produce two latent space representations $i_z, p_z \\in Z$ united by a joint 1024 dimensional vector space. In our `BASELINE` algorithm (Section 7) we apply the two encoders with no additional training. Although the two networks have achieved reasonable performance and robustness, motivated by several previous works [19][20][21] our aim is to fine-tune the two architectures in order to make them more proficient in our downstream task. Intuitively, applying transfer learning, our attempt is to build two encoders ($\\Psi^*, \\Gamma^*$)  which are trained in order to provide more refined text and image feature descriptions for our narrow domain. Ultimately, a more characterizing representation of visual and textual prompts leads straightforwardly to a more accurate image-text comparison and consequently a more conscious identification of the final bounding box. The adoption of a pretrained model, not only saves precious computational costs and allows us to benefit from state-of-the-art models without having to train one from scratch, but it also reduces the carbon footprint, which is one of the major concerns of contemporary scientific literature [22].\n",
        "\n",
        "As a first attempt to accomplish our objectives we have experimented with standard fine-tuning procedures that add multilayer perceptrons on top of the freezed pretrained architecture. Specifically we have designed, developed, trained and evaluated the following architectures:\n",
        "*   `net 1` (Figure 14). In this architecture we fine-tune only the image encoder module of CLIP while preserving the pretrained weights of the textual counterpart. We add a non linear activation function (ReLU) and a linear head on top of the 1024 CLIP features. The deep neural network has been trained with Stochastic Gradient Descent optimizer. The implementation of this architecture is reported in Section 10.1.\n",
        "*   `net 2` (Figure 15). In this second standard fine-tuned architecture, we add a bottleneck on top of the 1024 features proposed by CLIP. Having such a shrinkage encourages the network to compress feature representations to best fit in the available space. This time, both the image encoder and text encoder are fine-tuned symmetrically. The deep neural network has been trained with Stochastic Gradient Descent optimizer. The implementation of this architecture is reported in Section 10.1.\n",
        "* `net 3` (Figure 16). This third architecture resembles the previous one. We maintain the idea of the bottleneck but we use a different activation function in the text encoder portion of the network. The neurons of the natural language processing portion of the network are activated by a Sigmoid. Moreover, for the sake of experimenting as much as possible, this network is trained with Adagrad optimizer instead of SGD. The implementation of this architecture is reported in Section 10.1.\n",
        "* `net 4` (Figure 17). In this network we try to reduce the number of features of the final output layer. To accomplish this we append a linear head with 1024 input features and 512 output neurons on top of CLIP text and image encoders. As in the case of `net 1` we use the ReLU activation function and the SGD optimizer.\n",
        "\n",
        "Throughout the epochs we keep track of the loss and of the accuracy on the training and validation sets. Recalling that in this stage the algorithm should predict the bounding box proposed by DETR in step 1 characterized by the highest IoU with the dataset groundtruth, the most suitable loss function to be minimized is cross entropy loss. On the other hand, as a representative metric to quantify the accuracy in the predictions, we annotate the average intersection over union.\n",
        "\n",
        "We have trained each of these architectures on Azure GPUs (see Section 3) for 1.5 hours in order to figure out the effects of the various design choices. With the aim of saving precious computational resources but at the same time achieving a meaningful comparison of the proposed architectures, we have systematically limited the dataset. As described in Section 5, this last contains more than one annotation for a given image. Consequently, for the purpose of maximizing the number of images inspected by our apprentice software agent, we consider only one annotation for every image. Since the dataset is still too large for this first sequence of tests, we have further reduced its cardinality by selecting a random subset of training items.\n",
        "\n",
        "The obtained results are reported in the following tables and graphs. As we can see all the networks perform decently well and except `net 3` they all display ideal training curves. Looking at the loss and accuracy curves of the third standard fine tuned neural network, we clearly understand that Adagrad converges much slower than Stochastic Gradient Descent.\n",
        "From these preliminary tests, we notice that though the standard finetuning approach is reasonable, it seems inappropriate to achieve satisfactory improvements with respect to the `BASELINE` solution. In the following section we describe a contrastive learning based fine-tuning solution alternative which immediately displays more promising results.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZN33BCCFiJK"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/14.png)\n",
        "> \n",
        "> **Figure 14**\n",
        "> \n",
        "> `net1`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vu-DK_wfFjG8"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/15.png)\n",
        "> \n",
        "> **Figure 15**\n",
        "> \n",
        "> `net2`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LmqeelH0Fu9W"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/16.png)\n",
        "> \n",
        "> **Figure 16**\n",
        "> \n",
        "> `net3`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fO4vs5xRF3nh"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/17.png)\n",
        "> \n",
        "> **Figure 17**\n",
        "> \n",
        "> `net4`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ALozjEZc9ya6"
      },
      "source": [
        "> ![loss-net1.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/18.png)\n",
        "> \n",
        "> **Figure 18**\n",
        "> \n",
        "> `net1` loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gx1Qats7-Bci"
      },
      "source": [
        "> ![loss-net2.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/19.png)\n",
        "> \n",
        "> **Figure 19**\n",
        "> \n",
        "> `net2` loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LwgBGLeB-HFs"
      },
      "source": [
        "> ![loss-net3.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/20.png)\n",
        "> \n",
        "> **Figure 20**\n",
        "> \n",
        "> `net3` loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eMxRo1d-Nyq"
      },
      "source": [
        "> ![loss-net4.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/21.png)\n",
        "> \n",
        "> **Figure 21**\n",
        "> \n",
        "> `net4` loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mLz_lDTC-UPD"
      },
      "source": [
        "> ![acc-net1.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/22.png)\n",
        "> \n",
        "> **Figure 22**\n",
        "> \n",
        "> `net1` acc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmxsyApA-aRN"
      },
      "source": [
        "> ![acc-net2.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/23.png)\n",
        "> \n",
        "> **Figure 23**\n",
        "> \n",
        "> `net2` acc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lgwZn5GX-kpP"
      },
      "source": [
        "> ![acc-net3.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/24.png)\n",
        "> \n",
        "> **Figure 24**\n",
        "> \n",
        "> `net3` acc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pw8X4jNF-pCX"
      },
      "source": [
        "> ![acc-net4.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/25.png)\n",
        "> \n",
        "> **Figure 25**\n",
        "> \n",
        "> `net4` acc"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 10.1 Code"
      ],
      "metadata": {
        "id": "Mb9Bj6a8ejkm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipSfCore(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        img_encoder: nn.Module,\n",
        "        txt_encoder: nn.Module,\n",
        "    ):\n",
        "        super().__init__()\n",
        "        self.img_encoder = img_encoder\n",
        "        self.txt_encoder = txt_encoder\n",
        "\n",
        "    def cosine_similarity(\n",
        "        self,\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"],\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"],\n",
        "    ) -> Float[torch.Tensor, \"prompts crops\"]:\n",
        "        # normalise the image and the text\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"] = crops_z / crops_z.norm(dim=-1, keepdim=True)\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"] = prompts_z / prompts_z.norm(dim=-1, keepdim=True)\n",
        "\n",
        "        # evaluate the cosine similarity between the sets of features\n",
        "        return prompts_z @ crops_z.T\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        crops: Float[torch.Tensor, \"crops 3 244 244\"],\n",
        "        prompts: Int[torch.Tensor, \"prompts 77\"],\n",
        "    ) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        # step 1: compute crop representation in the latent space\n",
        "        crop_z: Float[torch.Tensor, \"crops 1024\"] = self.img_encoder(crops)\n",
        "\n",
        "        # step 2: compute prompt representation in the latent space\n",
        "        prompt_z: Int[torch.Tensor, \"prompts 1024\"] = self.txt_encoder(prompts)\n",
        "\n",
        "        # step 3: evaluate logits\n",
        "        similarity_matrix: Float[torch.Tensor, \"prompts crops\"] = self.cosine_similarity(crop_z, prompt_z)\n",
        "\n",
        "        # step 4: crops classification\n",
        "        return torch.mean(similarity_matrix, dim=0)"
      ],
      "metadata": {
        "id": "P9CIW4Lzei6c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipSf(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        img_encoder: nn.Module,\n",
        "        txt_encoder: nn.Module,\n",
        "    ):\n",
        "        super().__init__()\n",
        "        self.img_preprocess: Compose = preprocess\n",
        "        self.txt_preprocess: t.Callable[[t.Union[str, list[str]]], Float[torch.Tensor, \"77\"]] = clip.tokenize\n",
        "        self.core = ClipSfCore(img_encoder, txt_encoder)\n",
        "\n",
        "    def forward(self, crops: list[TensorImage], prompts: list[str]) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        # step 1: preprocess crops as required by the visual encoder\n",
        "        with torch.no_grad():\n",
        "            crops_preprocessed: Float[torch.Tensor, \"crops 3 244 244\"] = torch.stack([\n",
        "                self.img_preprocess(crop)\n",
        "                for crop in crops\n",
        "            ])\n",
        "\n",
        "        # step 2: preprocess prompts as required by the text encoder\n",
        "        with torch.no_grad():\n",
        "            prompts_preprocessed: Int[torch.Tensor, \"prompts 77\"] = self.txt_preprocess(prompts)\n",
        "\n",
        "        return self.core(crops_preprocessed, prompts_preprocessed)"
      ],
      "metadata": {
        "id": "wZz9ORNZenIe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "v9LxPAd_eqQf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "_ = lambda params: torch.optim.SGD(params=params, lr=.01, weight_decay=.01, momentum=.9)\n",
        "\n",
        "eval_summary(\n",
        "    ClipSf(\n",
        "        img_encoder=nn.Sequential(\n",
        "            clip_frozen_img_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 1024)\n",
        "        ),\n",
        "        txt_encoder=clip_frozen_txt_encoder,\n",
        "    ).to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ARTPdAQyerEQ",
        "outputId": "607c7591-bd31-43e0-c390-d3a7009a8962"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipSfCore                               [5, 3, 244, 244]          [5]                       --                        True\n",
              "\u251c\u2500Sequential: 1-1                        [5, 3, 244, 244]          [5, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenImgEnc: 2-1             [5, 3, 244, 244]          [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-2                         [5, 1024]                 [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-3                       [5, 1024]                 [5, 1024]                 1,049,600                 True\n",
              "\u251c\u2500ClipFrozenTxtEnc: 1-2                  [2, 77]                   [2, 1024]                 --                        --\n",
              "============================================================================================================================================\n",
              "Total params: 1,049,600\n",
              "Trainable params: 1,049,600\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 5.25\n",
              "============================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 0.04\n",
              "Params size (MB): 4.20\n",
              "Estimated Total Size (MB): 7.81\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "_ = lambda params: torch.optim.SGD(params=params, lr=.01, weight_decay=.01, momentum=.9)\n",
        "\n",
        "eval_summary(\n",
        "    ClipSf(\n",
        "        img_encoder=nn.Sequential(\n",
        "            clip_frozen_img_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 256),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(256, 1024),\n",
        "        ),\n",
        "        txt_encoder=nn.Sequential(\n",
        "            clip_frozen_txt_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 256),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(256, 1024),\n",
        "        ),\n",
        "    ).to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pveFK56Fes7W",
        "outputId": "f13f537e-7a1b-4ffe-e956-4d2c9fb9b4a1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipSfCore                               [5, 3, 244, 244]          [5]                       --                        True\n",
              "\u251c\u2500Sequential: 1-1                        [5, 3, 244, 244]          [5, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenImgEnc: 2-1             [5, 3, 244, 244]          [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-2                         [5, 1024]                 [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-3                       [5, 1024]                 [5, 512]                  524,800                   True\n",
              "\u2502    \u2514\u2500ReLU: 2-4                         [5, 512]                  [5, 512]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-5                       [5, 512]                  [5, 256]                  131,328                   True\n",
              "\u2502    \u2514\u2500ReLU: 2-6                         [5, 256]                  [5, 256]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-7                       [5, 256]                  [5, 1024]                 263,168                   True\n",
              "\u251c\u2500Sequential: 1-2                        [2, 77]                   [2, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenTxtEnc: 2-8             [2, 77]                   [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-9                         [2, 1024]                 [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-10                      [2, 1024]                 [2, 512]                  524,800                   True\n",
              "\u2502    \u2514\u2500ReLU: 2-11                        [2, 512]                  [2, 512]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-12                      [2, 512]                  [2, 256]                  131,328                   True\n",
              "\u2502    \u2514\u2500ReLU: 2-13                        [2, 256]                  [2, 256]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-14                      [2, 256]                  [2, 1024]                 263,168                   True\n",
              "============================================================================================================================================\n",
              "Total params: 1,838,592\n",
              "Trainable params: 1,838,592\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 6.44\n",
              "============================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 0.10\n",
              "Params size (MB): 7.35\n",
              "Estimated Total Size (MB): 11.03\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "_ = lambda params: torch.optim.Adadelta(params=params, lr=.0015, weight_decay=.01)\n",
        "\n",
        "eval_summary(\n",
        "    ClipSf(\n",
        "        img_encoder=nn.Sequential(\n",
        "            clip_frozen_img_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 256),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(256, 1024),\n",
        "        ),\n",
        "        txt_encoder=nn.Sequential(\n",
        "            clip_frozen_txt_encoder,\n",
        "            nn.Sigmoid(),\n",
        "            nn.Linear(1024, 512),\n",
        "            nn.Sigmoid(),\n",
        "            nn.Linear(512, 256),\n",
        "            nn.Sigmoid(),\n",
        "            nn.Linear(256, 1024),\n",
        "        ),\n",
        "    ).to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Obz65UFteuyp",
        "outputId": "8970eb9d-3dbe-4319-e755-8f404cf3d7e8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipSfCore                               [5, 3, 244, 244]          [5]                       --                        True\n",
              "\u251c\u2500Sequential: 1-1                        [5, 3, 244, 244]          [5, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenImgEnc: 2-1             [5, 3, 244, 244]          [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-2                         [5, 1024]                 [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-3                       [5, 1024]                 [5, 512]                  524,800                   True\n",
              "\u2502    \u2514\u2500ReLU: 2-4                         [5, 512]                  [5, 512]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-5                       [5, 512]                  [5, 256]                  131,328                   True\n",
              "\u2502    \u2514\u2500ReLU: 2-6                         [5, 256]                  [5, 256]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-7                       [5, 256]                  [5, 1024]                 263,168                   True\n",
              "\u251c\u2500Sequential: 1-2                        [2, 77]                   [2, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenTxtEnc: 2-8             [2, 77]                   [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Sigmoid: 2-9                      [2, 1024]                 [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-10                      [2, 1024]                 [2, 512]                  524,800                   True\n",
              "\u2502    \u2514\u2500Sigmoid: 2-11                     [2, 512]                  [2, 512]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-12                      [2, 512]                  [2, 256]                  131,328                   True\n",
              "\u2502    \u2514\u2500Sigmoid: 2-13                     [2, 256]                  [2, 256]                  --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-14                      [2, 256]                  [2, 1024]                 263,168                   True\n",
              "============================================================================================================================================\n",
              "Total params: 1,838,592\n",
              "Trainable params: 1,838,592\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 6.44\n",
              "============================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 0.10\n",
              "Params size (MB): 7.35\n",
              "Estimated Total Size (MB): 11.03\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "_ = lambda params: torch.optim.SGD(params=params, lr=.01, weight_decay=.01, momentum=.9)\n",
        "\n",
        "eval_summary(\n",
        "    ClipSf(\n",
        "        img_encoder=nn.Sequential(\n",
        "            clip_frozen_img_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "        ),\n",
        "        txt_encoder=nn.Sequential(\n",
        "            clip_frozen_txt_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "        ),\n",
        "    ).to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QGhje8_Vewhi",
        "outputId": "206cd785-ab30-406a-c8f9-206fba0e632b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipSfCore                               [5, 3, 244, 244]          [5]                       --                        True\n",
              "\u251c\u2500Sequential: 1-1                        [5, 3, 244, 244]          [5, 512]                  --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenImgEnc: 2-1             [5, 3, 244, 244]          [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-2                         [5, 1024]                 [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-3                       [5, 1024]                 [5, 512]                  524,800                   True\n",
              "\u251c\u2500Sequential: 1-2                        [2, 77]                   [2, 512]                  --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenTxtEnc: 2-4             [2, 77]                   [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-5                         [2, 1024]                 [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-6                       [2, 1024]                 [2, 512]                  524,800                   True\n",
              "============================================================================================================================================\n",
              "Total params: 1,049,600\n",
              "Trainable params: 1,049,600\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 3.67\n",
              "============================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 0.03\n",
              "Params size (MB): 4.20\n",
              "Estimated Total Size (MB): 7.80\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "HLKBtEHqe0BE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "loss_fn: t.Callable[[Float[torch.Tensor, \"crops\"], Int[torch.Tensor, \"1\"]], Float[torch.Tensor, \"1\"]] = nn.functional.cross_entropy"
      ],
      "metadata": {
        "id": "CpmhlfE-ezpC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def training_step(\n",
        "        model: nn.Module,\n",
        "        data_loader: DataLoader[\n",
        "            tuple[\n",
        "                tuple[TensorImage, ...],\n",
        "                tuple[str, ...],\n",
        "                int,\n",
        "                Float[torch.Tensor, \"crops 4\"],\n",
        "                Float[torch.Tensor, \"1 4\"],\n",
        "            ]\n",
        "        ],\n",
        "        optimizer: torch.optim.Optimizer,\n",
        ") -> tuple[float, float]:\n",
        "    model.train()\n",
        "\n",
        "    running_loss: float = 0\n",
        "    running_acc: float = 0\n",
        "    progress = tqdm(data_loader, desc=\"training\")\n",
        "\n",
        "    cropss: tuple[tuple[TensorImage, ...], ...]\n",
        "    promptss: tuple[tuple[str, ...], ...]\n",
        "    true_is: tuple[int, ...]\n",
        "    xyxyss: tuple[Float[torch.Tensor, \"crops 4\"], ...]\n",
        "    true_xyxys: tuple[Float[torch.Tensor, \"1 4\"], ...]\n",
        "\n",
        "    for iter, (cropss, promptss, true_is, xyxyss, true_xyxys) in zip(it.count(1), progress):\n",
        "        # forward pass\n",
        "        preds: list[Float[torch.Tensor, \"crops\"]] = [\n",
        "            model(crops, prompts) for crops, prompts in zip(cropss, promptss)\n",
        "        ]\n",
        "\n",
        "        # calculate loss\n",
        "        losses: Float[torch.Tensor, \"batch\"] = torch.stack([\n",
        "            loss_fn(pred, torch.tensor(true_i))\n",
        "            for pred, true_i in zip(preds, true_is)\n",
        "        ])\n",
        "        loss: Float[torch.Tensor, \"1\"] = torch.mean(losses)\n",
        "        running_loss += loss.item()\n",
        "\n",
        "        # optimizer zero grad\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # loss backward\n",
        "        loss.backward()\n",
        "\n",
        "        # optimizer step\n",
        "        optimizer.step()\n",
        "\n",
        "        # calculate IoU accuracy\n",
        "        with torch.inference_mode():\n",
        "            # # get indexes of the predicted bounding box to compute IoU accuracy\n",
        "            pred_is: list[int] = [\n",
        "                torch.argmax(pred).item()\n",
        "                for pred in preds\n",
        "            ]\n",
        "\n",
        "            # # get predicted bounding boxes\n",
        "            pred_xyxys: list[Float[torch.Tensor, \"4\"]] = [\n",
        "                xyxys[pred_i]\n",
        "                for xyxys, pred_i in zip(xyxyss, pred_is)\n",
        "            ]\n",
        "\n",
        "            # # IoU\n",
        "            acc: float = torch.mean(box_iou(torch.cat(true_xyxys), torch.stack(pred_xyxys)).diagonal()).item()\n",
        "            running_acc += acc\n",
        "\n",
        "            progress.set_postfix(\n",
        "                {\n",
        "                    \"loss\": running_loss / iter,\n",
        "                    \"iou\": running_acc / iter,\n",
        "                },\n",
        "                refresh=False,\n",
        "            )\n",
        "\n",
        "    return running_loss / len(data_loader), running_acc / len(data_loader)"
      ],
      "metadata": {
        "id": "p6Em7Mp9e2X6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def test_step(\n",
        "        model: nn.Module,\n",
        "        data_loader: DataLoader[tuple[TensorImage, list[str], Float[torch.Tensor, \"X 4\"], Float[torch.Tensor, \"4\"]]],\n",
        ") -> tuple[float, float]:\n",
        "    model.eval()\n",
        "\n",
        "    running_loss: float = 0\n",
        "    running_acc: float = 0\n",
        "    progress = tqdm(data_loader, desc=\"testing\")\n",
        "\n",
        "    with torch.inference_mode():\n",
        "        img: TensorImage\n",
        "        prompts: list[str]\n",
        "        xyxys: Float[torch.Tensor, \"crops 4\"]\n",
        "        xyxy: Float[torch.Tensor, \"4\"]\n",
        "\n",
        "        for iter, (img, prompts, xyxys, true_xyxy) in zip(it.count(1), progress):\n",
        "            true_i: int = best_bbox(xyxys, true_xyxy)\n",
        "\n",
        "            # from xyxys to crops\n",
        "            xywhs: Int[torch.Tensor, \"X 4\"] = box_convert(xyxys, in_fmt=\"xyxy\", out_fmt=\"xywh\").round().int()\n",
        "\n",
        "            crops: list[TensorImage] = [\n",
        "                crop(img, top=y, left=x, height=h, width=w)\n",
        "                for xywh in xywhs\n",
        "                for [x, y, w, h] in [xywh.tolist()]\n",
        "            ]\n",
        "\n",
        "            # forward pass\n",
        "            model_output: Float[torch.Tensor, \"crops\"] = model(crops, prompts)\n",
        "\n",
        "            # calculate loss\n",
        "            loss: float = loss_fn(model_output, torch.tensor(true_i)).item()\n",
        "            running_loss += loss\n",
        "\n",
        "            # calculate IoU accuracy\n",
        "\n",
        "            # # get index of the predicted bounding box to compute IoU accuracy\n",
        "            pred_i: int = torch.argmax(model_output).item()\n",
        "\n",
        "            # # get predicted bounding\n",
        "            pred_xyxy: Float[torch.Tensor, \"4\"] = xyxys[pred_i]\n",
        "\n",
        "            # # IoU\n",
        "            acc: float = box_iou(true_xyxy, pred_xyxy.unsqueeze(0)).item()\n",
        "            running_acc += acc\n",
        "\n",
        "            progress.set_postfix(\n",
        "                {\n",
        "                    \"loss\": running_loss / iter,\n",
        "                    \"iou\": running_acc / iter,\n",
        "                },\n",
        "                refresh=False,\n",
        "            )\n",
        "\n",
        "        return running_loss / len(data_loader), running_acc / len(data_loader)"
      ],
      "metadata": {
        "id": "7eMSZLfke4C7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "HVMauPDFe55A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "keys: list[str] = [f\"net{i + 1}\" for i in range(4)]"
      ],
      "metadata": {
        "id": "TCvqXYzSe64_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.concat(\n",
        "    [\n",
        "        pd.read_csv(f\"assets/standard-finetuning/eval-{key}.csv\", index_col=0)\n",
        "        for key in keys\n",
        "    ],\n",
        "    axis=1,\n",
        "    keys=keys,\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "id": "M3ecuu_Ae9Ce",
        "outputId": "449c5134-8c57-4925-e743-67c7baf95601"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              net1                                           net2  \\\n",
              "               iou cos similarity euclidean distance          iou   \n",
              "count  5023.000000    5023.000000        5023.000000  5023.000000   \n",
              "mean      0.457677       0.852569           0.954881     0.380285   \n",
              "std       0.412066       0.140639           0.531035     0.396900   \n",
              "min       0.000000       0.285263           0.000000     0.000000   \n",
              "25%       0.028989       0.757906           0.445398     0.016817   \n",
              "50%       0.325832       0.895899           0.932816     0.178222   \n",
              "75%       0.935840       0.976137           1.367539     0.901817   \n",
              "max       0.996050       1.000000           2.881575     0.998143   \n",
              "\n",
              "                                                net3                 \\\n",
              "      cos similarity euclidean distance          iou cos similarity   \n",
              "count    5023.000000        5023.000000  5023.000000    5023.000000   \n",
              "mean        0.813678           1.112312     0.355783       0.804044   \n",
              "std         0.157147           0.576089     0.390836       0.158404   \n",
              "min         0.276203           0.000000     0.000000       0.259929   \n",
              "25%         0.698765           0.528522     0.006893       0.688604   \n",
              "50%         0.839692           1.153889     0.148990       0.825239   \n",
              "75%         0.967429           1.566146     0.867523       0.961532   \n",
              "max         1.000000           2.881575     0.996050       1.000000   \n",
              "\n",
              "                                 net4                                    \n",
              "      euclidean distance          iou cos similarity euclidean distance  \n",
              "count        5023.000000  5023.000000    5023.000000        5023.000000  \n",
              "mean            1.150898     0.405283       0.833840           1.026658  \n",
              "std             0.575043     0.408287       0.145446           0.542099  \n",
              "min             0.000000     0.000000       0.290687           0.000000  \n",
              "25%             0.575107     0.007908       0.731535           0.482025  \n",
              "50%             1.205171     0.202125       0.865183           1.051583  \n",
              "75%             1.606206     0.921305       0.971797           1.455355  \n",
              "max             2.998202     0.998143       1.000000           2.559178  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-55d43e0d-8573-4820-a252-65ce2d8f8946\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"3\" halign=\"left\">net1</th>\n",
              "      <th colspan=\"3\" halign=\"left\">net2</th>\n",
              "      <th colspan=\"3\" halign=\"left\">net3</th>\n",
              "      <th colspan=\"3\" halign=\"left\">net4</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.457677</td>\n",
              "      <td>0.852569</td>\n",
              "      <td>0.954881</td>\n",
              "      <td>0.380285</td>\n",
              "      <td>0.813678</td>\n",
              "      <td>1.112312</td>\n",
              "      <td>0.355783</td>\n",
              "      <td>0.804044</td>\n",
              "      <td>1.150898</td>\n",
              "      <td>0.405283</td>\n",
              "      <td>0.833840</td>\n",
              "      <td>1.026658</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.412066</td>\n",
              "      <td>0.140639</td>\n",
              "      <td>0.531035</td>\n",
              "      <td>0.396900</td>\n",
              "      <td>0.157147</td>\n",
              "      <td>0.576089</td>\n",
              "      <td>0.390836</td>\n",
              "      <td>0.158404</td>\n",
              "      <td>0.575043</td>\n",
              "      <td>0.408287</td>\n",
              "      <td>0.145446</td>\n",
              "      <td>0.542099</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.285263</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.276203</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.259929</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.290687</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.028989</td>\n",
              "      <td>0.757906</td>\n",
              "      <td>0.445398</td>\n",
              "      <td>0.016817</td>\n",
              "      <td>0.698765</td>\n",
              "      <td>0.528522</td>\n",
              "      <td>0.006893</td>\n",
              "      <td>0.688604</td>\n",
              "      <td>0.575107</td>\n",
              "      <td>0.007908</td>\n",
              "      <td>0.731535</td>\n",
              "      <td>0.482025</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.325832</td>\n",
              "      <td>0.895899</td>\n",
              "      <td>0.932816</td>\n",
              "      <td>0.178222</td>\n",
              "      <td>0.839692</td>\n",
              "      <td>1.153889</td>\n",
              "      <td>0.148990</td>\n",
              "      <td>0.825239</td>\n",
              "      <td>1.205171</td>\n",
              "      <td>0.202125</td>\n",
              "      <td>0.865183</td>\n",
              "      <td>1.051583</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.935840</td>\n",
              "      <td>0.976137</td>\n",
              "      <td>1.367539</td>\n",
              "      <td>0.901817</td>\n",
              "      <td>0.967429</td>\n",
              "      <td>1.566146</td>\n",
              "      <td>0.867523</td>\n",
              "      <td>0.961532</td>\n",
              "      <td>1.606206</td>\n",
              "      <td>0.921305</td>\n",
              "      <td>0.971797</td>\n",
              "      <td>1.455355</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.996050</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.881575</td>\n",
              "      <td>0.998143</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.881575</td>\n",
              "      <td>0.996050</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.998202</td>\n",
              "      <td>0.998143</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.559178</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-55d43e0d-8573-4820-a252-65ce2d8f8946')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-55d43e0d-8573-4820-a252-65ce2d8f8946 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-55d43e0d-8573-4820-a252-65ce2d8f8946');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-47c26c35-bab1-475e-86dd-89271bc589f2\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-47c26c35-bab1-475e-86dd-89271bc589f2')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-47c26c35-bab1-475e-86dd-89271bc589f2 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 4**"
      ],
      "metadata": {
        "id": "8KHPDhL8PUp_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.read_csv(\"assets/standard-finetuning/comparing.csv\", index_col=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "UKwIeHXOe-wW",
        "outputId": "336b907a-0b0c-4b07-d72b-e082fe3fb120"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      mAP[IoU .3]  mAP[IoU .5]  mAP[IoU .7]      mIoU      mCos       mED\n",
              "net1     0.511248     0.445550     0.401354  0.457677  0.852569  0.954881\n",
              "net2     0.414692     0.348995     0.314155  0.380285  0.813678  1.112312\n",
              "net3     0.386024     0.323512     0.290066  0.355783  0.804044  1.150898\n",
              "net4     0.448935     0.385626     0.347999  0.405283  0.833840  1.026658"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b163d871-f859-4f0b-8e6a-a3756e048203\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mAP[IoU .3]</th>\n",
              "      <th>mAP[IoU .5]</th>\n",
              "      <th>mAP[IoU .7]</th>\n",
              "      <th>mIoU</th>\n",
              "      <th>mCos</th>\n",
              "      <th>mED</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>net1</th>\n",
              "      <td>0.511248</td>\n",
              "      <td>0.445550</td>\n",
              "      <td>0.401354</td>\n",
              "      <td>0.457677</td>\n",
              "      <td>0.852569</td>\n",
              "      <td>0.954881</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>net2</th>\n",
              "      <td>0.414692</td>\n",
              "      <td>0.348995</td>\n",
              "      <td>0.314155</td>\n",
              "      <td>0.380285</td>\n",
              "      <td>0.813678</td>\n",
              "      <td>1.112312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>net3</th>\n",
              "      <td>0.386024</td>\n",
              "      <td>0.323512</td>\n",
              "      <td>0.290066</td>\n",
              "      <td>0.355783</td>\n",
              "      <td>0.804044</td>\n",
              "      <td>1.150898</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>net4</th>\n",
              "      <td>0.448935</td>\n",
              "      <td>0.385626</td>\n",
              "      <td>0.347999</td>\n",
              "      <td>0.405283</td>\n",
              "      <td>0.833840</td>\n",
              "      <td>1.026658</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b163d871-f859-4f0b-8e6a-a3756e048203')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b163d871-f859-4f0b-8e6a-a3756e048203 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b163d871-f859-4f0b-8e6a-a3756e048203');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f4193470-e169-47ff-9dd5-cc26630000b0\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f4193470-e169-47ff-9dd5-cc26630000b0')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f4193470-e169-47ff-9dd5-cc26630000b0 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 5**"
      ],
      "metadata": {
        "id": "qGxzf_ADPWQb"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRJFbhsjp8S8"
      },
      "source": [
        "## 11 Fine-tune like you pretrain\n",
        "\n",
        "`FLYP`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4ZWXK0zhyEvV"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir ./assets/flyp/runs --port 6003"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UYKerVcW5WQx"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir ./assets/flyp-solve-overfitting/runs --port 6004"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zhIscsM7zCo8"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir ./assets/flyp-optuna/runs --port 6005"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "28H3CEJKzE0z"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir ./assets/flyp-augmented/runs --port 6006"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9MEc6j2oy47"
      },
      "source": [
        "One of the trickiest aspects of standard fine-tuning approaches lies in the understanding of the role of the subtle applicable changes since there is no simple rule of thumb for what is the correct modification. As an alternative methodology to standard fine-tuning techniques, inspired by the work of Sachin Goyal et al. \"Finetune like your pretrain: Improved finetuning of zero-shot vision models\" [5], we propose to finetune the visual and textual encoder components of CLIP via the same pretraining process employed by the original authors of the model.\n",
        "\n",
        "More precisely, the purpose of CLIP is to learn a multi-modal embedding of image and text.\n",
        "\n",
        "Let $\\Psi: I \\rightarrow {R}^d$ denote the image encoder that maps an image to a $d$-dimensional image-text embedding space. $\\Psi$ is parameterized by parameters $\\theta_{\\text{img}}$.\n",
        "\n",
        "Let ${P}$ be the space for text descriptions of images. Analogously, $\\Gamma: {P} \\rightarrow {R}$  is the language encoder with model parameters $\\theta_{\\text{text}}$.\n",
        "\n",
        "The backbone of the pretraining objective is contrastive learning, where the goal is to align the embedding $\\Psi(I_i)$ of an image close to the embedding $\\Gamma(P_i)$ of its corresponding text description, and away from other text embeddings $\\Gamma(P_j)$ in the batch.\n",
        "\n",
        "Given a batch with $B$ images with their corresponding text descriptions $D = \\{(I_1, P_1), \\ldots, (I_B, P_B)\\}$, pretraining objective is formally formulated as follows:\n",
        "\n",
        "$$\n",
        "{L}_{\\text{pre}}(D, \\theta) :=\n",
        "\\sum_{i=1}^{B} -\\log{\\frac{\\exp(\\bar{f}(I_i) \\cdot \\bar{g}(P_i))}{\\sum_{j=1}^{B}\\exp(\\bar{f}(I_i) \\cdot \\bar{g}(P_j))}} + \\sum_{i=1}^{B} -\\log{\\frac{\\exp(\\bar{f}(I_i) \\cdot \\bar{g}(P_i))}{\\sum_{j=1}^{B}\\exp(\\bar{f}(I_j) \\cdot \\bar{g}(P_i))}}\n",
        "$$\n",
        "\n",
        "where\n",
        "\n",
        "- $\\theta = [\\theta_{\\text{img}}, \\theta_{\\text{text}}]$ are image and text encoder parameters\n",
        "- $\\bar{f}$ and $\\bar{g}$ are the $l_2$ normalized versions of $f$ and $g$ respectively.\n",
        "\n",
        "In other words, as a result of the minimization of this objective function, the algorithm maximizes the cosine similarity of the image and text embeddings of the $N$ real pairs in the batch while minimizing the cosine similarity of the embeddings of the $N^2-N$ incorrect pairings.\n",
        "\n",
        "In consonance with this promising direction we have designed, developed and trained a contrastive learning architecture as summarized in Figure 26. As depicted in the proposed illustration, we append a symmetric linear head on both the pre-trained CLIP encoders, where to accommodate our trainable parameters. Moreover, for the purpose of training the architecture we have implemented a second customized dataset class together with the corresponding data loaders. Indeed, the input and output of our network to be trained are significantly changed. In order to successfully apply contrastive learning, at each iteration we need to compare a collection of $B \\in {N}$ ground truth bounding boxes with their corresponding $B$ textual descriptions. Throughout the training epochs the goal is to maximize the cosine similarity between the crop-text couples which actually occur in the batch while minimizing the cosine similarity of the other pairs. To this end we have personalized the loss function proposed by [this GitHub repository](https://github.com/locuslab/FLYP/) whose code has been made available by the authors of the aforementioned \"Finetune like you pretrain\" paper [5]. On the other hand, unfortunately, in [the official GitHub repository of CLIP](https://github.com/OpenAI/CLIP) there is not the implementation of the original procedure used to train the model. Fortunately, we have been able to personalize for our scopes the code published at [this GitHub repository](https://github.com/mlfoundations/open_clip/) whose goal is to enable the training of models with contrastive image-text supervision. At the end of the day, the implementation follows closely the pseudocode proposed in the paper of CLIP that we conveniently report here:\n",
        "\n",
        "```python\n",
        "# image_encoder - ResNet or Vision Transformer\n",
        "# text_encoder - CBOW or Text Transformer\n",
        "# I[n, h, w, c] - minibatch of aligned images\n",
        "# T[n, l] - minibatch of aligned texts\n",
        "# W_i[d_i, d_e] - learned proj of image to embed\n",
        "# W_t[d_t, d_e] - learned proj of text to embed\n",
        "# t - learned temperature parameter\n",
        "\n",
        "# extract feature representations of each modality\n",
        "I_f = image_encoder(I) #[n, d_i]\n",
        "T_f = text_encoder(T)  #[n, d_t]\n",
        "\n",
        "# joint multimodal embedding [n, d_e]\n",
        "I_e = l2_normalize(np.dot(I_f, W_i), axis=1)\n",
        "T_e = l2_normalize(np.dot(T_f, W_t), axis=1)\n",
        "\n",
        "# scaled pairwise cosine similarities [n, n]\n",
        "logits = np.dot(I_e, T_e.T) * np.exp(t)\n",
        "\n",
        "# symmetric loss function\n",
        "labels = np.arange(n)\n",
        "loss_i = cross_entropy_loss(logits, labels, axis=0)\n",
        "loss_t = cross_entropy_loss(logits, labels, axis=1)\n",
        "loss = (loss_i + loss_t)/2\n",
        "```\n",
        "\n",
        "The resulting training procedure is extremely computationally efficient both in terms of time and memory consumption. Consequently, we have been able to train our fine-tuned image and text encoders on the whole dataset for many epochs in a reasonable amount of time. Also in this case, stochastic gradient descent has been the optimizer of choice. As regards the cardinality of the batches, contrastive learning procedures tend to provide better outcomes with larger batch sizes [23]. Intuitively, considering only one (image, text) pair at a time, i.e. a batch size of 1, the resulting contrastive loss is completely meaningless. Remarkably, with our proposed implementation we have succeeded in using a batch size of 1024 (image, text) pairs without exceeding memory limits.\n",
        "\n",
        "Once the two multi-modal encoders $\\Psi^*, \\Gamma^*$ have been symmetrically finetuned by the above-mentioned training strategy, we have tested their acquired skills on our downstream task. The obtained results are reported at the end of this section. The performance obtained by the model is impressive and outperforms all the previously proposed standard finetuning networks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghLgczrMGNgM"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/26.png)\n",
        "> \n",
        "> **Figure 26**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mpd9jFHQ7gPz"
      },
      "source": [
        "> ![Screenshot 2023-09-01 at 01.23.15.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/27.png)\n",
        "> \n",
        "> **Figure 27**\n",
        "> \n",
        "> `FLYP` loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SRf7OP99o8WM"
      },
      "source": [
        "### 11.1 Observations\n",
        "\n",
        "`FLYP solve overfitting`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gVSax1Zuo-S5"
      },
      "source": [
        "Looking at the training curves in the plot depicted in Figure 27, we can evidently understand that during training the model has overfitted around epoch 10. Gladly, the displayed behavior strongly suggests that there is still space for improvement with this architecture.\n",
        "\n",
        "Pursuing our objectives, we have refined the neural network and the training procedure by incorporating regularization techniques in order to prevent overfitting. Specifically, in our improved implementation we consider the following strategies.\n",
        "early stopping: we save the state of the network at the end of every epoch. Notably, the tweak poorly affects the speed of the computation. Following this strategy, after training, we can manually inspect the training and validation error and easily select the most optimal version of the model.\n",
        "dropout: we add a dropout that randomly shuts down some fraction of the layers' neurons at each training step by zeroing out their values.\n",
        "\n",
        "The effects of the application of these effective regularization strategies can be appreciated by the reader at the end of this subsection. Manifestly, the training curves now exhibit an optimal descending behavior. The encouraging trend is visually reflected in the matrices shown in Figure 43, Figure 44 and Figure 45 in which the color of the diagonal becomes progressively brighter throughout the training process.\n",
        "\n",
        "Consistently with the improvements in the training objective, also the evaluation on the visual grounding downstream task of interest has produced astonishing results as reported in Table 10.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXaS2kk70Bsj"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/28.png)\n",
        "> \n",
        "> **Figure 28**\n",
        "> \n",
        "> Baseline prediction.\n",
        "> \n",
        "> *a chili dog with slices of cheese visible under the chili.*\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwv5wQBq0R_M"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/29.png)\n",
        "> \n",
        "> **Figure 29**\n",
        "> \n",
        "> Contrastive learning model prediction.\n",
        "> \n",
        "> *a chili dog with slices of cheese visible under the chili.*\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "jY0XQK6i2cUQ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TZWPkf9u2g-n"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/30.png)\n",
        "> \n",
        "> **Figure 30**\n",
        "> \n",
        "> \"train\" split"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vYo3aFGH2sF_"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/31.png)\n",
        "> \n",
        "> **Figure 31**\n",
        "> \n",
        "> \"test\" split"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OcHgR3xX21vi"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/32.png)\n",
        "> \n",
        "> **Figure 32**\n",
        "> \n",
        "> \"val\" split"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MyfADNfa7VrE"
      },
      "source": [
        "> ![Screenshot 2023-09-01 at 01.23.36.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/33.png)\n",
        "> \n",
        "> **Figure 33**\n",
        "> \n",
        "> `FLYP solve overfitting`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 11.2 Code"
      ],
      "metadata": {
        "id": "JsJQ_UD9fLrO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipFlypCore(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.img_encoder = nn.Sequential(\n",
        "            clip_frozen_img_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Dropout(.25),\n",
        "            nn.Linear(1024, 1024),\n",
        "        )\n",
        "        self.txt_encoder = nn.Sequential(\n",
        "            clip_frozen_txt_encoder,\n",
        "            nn.ReLU(),\n",
        "            nn.Dropout(.25),\n",
        "            nn.Linear(1024, 1024),\n",
        "        )\n",
        "        # the temperature parameter is added as suggested by the original paper in order to prevent training instability\n",
        "        self.logit_scale: Float[torch.Tensor, \"1\"] = nn.Parameter(\n",
        "            torch.log(torch.tensor(1 / 0.07))\n",
        "        )\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        crop: Float[torch.Tensor, \"entries 3 244 244\"],\n",
        "        prompt: Int[torch.Tensor, \"entries 77\"],\n",
        "    ) -> tuple[\n",
        "        Float[torch.Tensor, \"1024\"],\n",
        "        Float[torch.Tensor, \"1024\"],\n",
        "        Float[torch.Tensor, \"entries\"],\n",
        "    ]:\n",
        "        # step 1: compute crop representation in the latent space\n",
        "        crop_z: Float[torch.Tensor, \"entries 1024\"] = self.img_encoder(crop)\n",
        "\n",
        "        # step 2: compute prompt representation in the latent space\n",
        "        prompt_z: Int[torch.Tensor, \"entries 1024\"] = self.txt_encoder(prompt)\n",
        "\n",
        "        return crop_z, prompt_z, self.logit_scale.exp()"
      ],
      "metadata": {
        "id": "7yPlVg8HfNXu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipFlyp(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.img_preprocess: Compose = preprocess\n",
        "        self.txt_preprocess: t.Callable[[t.Union[str, list[str]]], Float[torch.Tensor, \"77\"]] = clip.tokenize\n",
        "        self.core = ClipFlypCore()\n",
        "\n",
        "    def forward(\n",
        "        self, entries: list[tuple[TensorImage, str]]\n",
        "    ) -> Float[torch.Tensor, \"entries\"]:\n",
        "        # step 1: preprocess crops as required by the visual encoder\n",
        "        with torch.no_grad():\n",
        "            crops_preprocessed: Float[torch.Tensor, \"entries 3 244 244\"] = torch.stack([\n",
        "                self.img_preprocess(crop)\n",
        "                for crop, _ in entries\n",
        "            ])\n",
        "\n",
        "        # step 2: preprocess prompts as required by the text encoder\n",
        "        with torch.no_grad():\n",
        "            prompts_preprocessed: Int[torch.Tensor, \"entries 77\"] = self.txt_preprocess([\n",
        "                prompt\n",
        "                for _, prompt in entries\n",
        "            ])\n",
        "\n",
        "        return self.core(crops_preprocessed, prompts_preprocessed)"
      ],
      "metadata": {
        "id": "69dB23QyfPhf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "_ = lambda params: torch.optim.SGD(params=params, lr=.01, weight_decay=.01, momentum=.9)\n",
        "_ = lambda params: torch.optim.Adam(params=params, lr=.00043, weight_decay=.01)\n",
        "\n",
        "contrastive_summary(\n",
        "    ClipFlyp().to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K1GHDGTtfRU5",
        "outputId": "7469db6f-70d8-4cbd-9b73-538460d55bca"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipFlypCore                             [8, 3, 244, 244]          [8, 1024]                 1                         True\n",
              "\u251c\u2500Sequential: 1-1                        [8, 3, 244, 244]          [8, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenImgEnc: 2-1             [8, 3, 244, 244]          [8, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-2                         [8, 1024]                 [8, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Dropout: 2-3                      [8, 1024]                 [8, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-4                       [8, 1024]                 [8, 1024]                 1,049,600                 True\n",
              "\u251c\u2500Sequential: 1-2                        [8, 77]                   [8, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenTxtEnc: 2-5             [8, 77]                   [8, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-6                         [8, 1024]                 [8, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Dropout: 2-7                      [8, 1024]                 [8, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-8                       [8, 1024]                 [8, 1024]                 1,049,600                 True\n",
              "============================================================================================================================================\n",
              "Total params: 2,099,201\n",
              "Trainable params: 2,099,201\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 16.79\n",
              "============================================================================================================================================\n",
              "Input size (MB): 5.72\n",
              "Forward/backward pass size (MB): 0.13\n",
              "Params size (MB): 8.40\n",
              "Estimated Total Size (MB): 14.25\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "f28E1InMfTmo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipFlypEvalCore(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        img_encoder: nn.Module,\n",
        "        txt_encoder: nn.Module,\n",
        "    ):\n",
        "        super().__init__()\n",
        "        self.img_encoder = img_encoder\n",
        "        self.txt_encoder = txt_encoder\n",
        "\n",
        "    def cosine_similarity(\n",
        "        self,\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"],\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"],\n",
        "    ) -> Float[torch.Tensor, \"prompts crops\"]:\n",
        "        # normalise the image and the text\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"] = crops_z / crops_z.norm(dim=-1, keepdim=True)\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"] = prompts_z / prompts_z.norm(dim=-1, keepdim=True)\n",
        "\n",
        "        # evaluate the cosine similarity between the sets of features\n",
        "        return prompts_z @ crops_z.T\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        crops: Float[torch.Tensor, \"crops 3 244 244\"],\n",
        "        prompts: Int[torch.Tensor, \"prompts 77\"],\n",
        "    ) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        # step 1: compute crop representation in the latent space\n",
        "        crop_z: Float[torch.Tensor, \"crops 1024\"] = self.img_encoder(crops)\n",
        "\n",
        "        # step 2: compute prompt representation in the latent space\n",
        "        prompt_z: Int[torch.Tensor, \"prompts 1024\"] = self.txt_encoder(prompts)\n",
        "\n",
        "        # step 3: evaluate logits\n",
        "        similarity_matrix: Float[torch.Tensor, \"prompts crops\"] = self.cosine_similarity(crop_z, prompt_z)\n",
        "\n",
        "        # step 4: crops classification\n",
        "        return torch.mean(similarity_matrix, dim=0)"
      ],
      "metadata": {
        "id": "VuO2eSzyfUdt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipFlypEval(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        img_encoder: nn.Module,  # visual encoder\n",
        "        txt_encoder: nn.Module,  # natural language prompts encoder\n",
        "    ):\n",
        "        super().__init__()\n",
        "        self.img_preprocess: Compose = preprocess\n",
        "        self.txt_preprocess: t.Callable[[t.Union[str, list[str]]], Float[torch.Tensor, \"77\"]] = clip.tokenize\n",
        "        self.core = ClipFlypEvalCore(img_encoder, txt_encoder)\n",
        "\n",
        "    def forward(\n",
        "        self, crops: list[TensorImage], prompts: list[str]\n",
        "    ) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        # step 1: preprocess crops as required by the visual encoder\n",
        "        with torch.no_grad():\n",
        "            crops_preprocessed: Float[torch.Tensor, \"crops 3 244 244\"] = torch.stack([\n",
        "                self.img_preprocess(crop)\n",
        "                for crop in crops\n",
        "            ])\n",
        "\n",
        "        # step 2: preprocess prompts as required by the text encoder\n",
        "        with torch.no_grad():\n",
        "            prompts_preprocessed: Int[torch.Tensor, \"prompts 77\"] = self.txt_preprocess(prompts)\n",
        "\n",
        "        return self.core(crops_preprocessed, prompts_preprocessed)"
      ],
      "metadata": {
        "id": "1C4NexmJfWLi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_summary(\n",
        "    ClipFlypEval(\n",
        "        ClipFlyp().core.img_encoder,\n",
        "        ClipFlyp().core.txt_encoder,\n",
        "    ).to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wszUZeLIfXmu",
        "outputId": "221a5802-e90c-405f-8d76-76d7cf4df39c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipFlypEvalCore                         [5, 3, 244, 244]          [5]                       --                        True\n",
              "\u251c\u2500Sequential: 1-1                        [5, 3, 244, 244]          [5, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenImgEnc: 2-1             [5, 3, 244, 244]          [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-2                         [5, 1024]                 [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Dropout: 2-3                      [5, 1024]                 [5, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-4                       [5, 1024]                 [5, 1024]                 1,049,600                 True\n",
              "\u251c\u2500Sequential: 1-2                        [2, 77]                   [2, 1024]                 --                        True\n",
              "\u2502    \u2514\u2500ClipFrozenTxtEnc: 2-5             [2, 77]                   [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500ReLU: 2-6                         [2, 1024]                 [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Dropout: 2-7                      [2, 1024]                 [2, 1024]                 --                        --\n",
              "\u2502    \u2514\u2500Linear: 2-8                       [2, 1024]                 [2, 1024]                 1,049,600                 True\n",
              "============================================================================================================================================\n",
              "Total params: 2,099,200\n",
              "Trainable params: 2,099,200\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 7.35\n",
              "============================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 0.06\n",
              "Params size (MB): 8.40\n",
              "Estimated Total Size (MB): 12.03\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "TXF9QWUMffy3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipLoss(nn.Module):\n",
        "    def forward(\n",
        "        self,\n",
        "        imgs_features: Float[torch.Tensor, \"entries 1024\"],\n",
        "        txts_features: Float[torch.Tensor, \"entries 1024\"],\n",
        "        logit_scale: Float[torch.Tensor, \"1\"],\n",
        "    ) -> Float[torch.Tensor, \"1\"]:\n",
        "        # compute logits per image and logits per text\n",
        "        logits_per_image: Float[torch.Tensor, \"entries entries\"] = logit_scale * imgs_features @ txts_features.T\n",
        "        logits_per_text: Float[torch.Tensor, \"entries entries\"] = logit_scale * txts_features @ imgs_features.T\n",
        "\n",
        "        # get ground truth labels for the computation of the cross entropy loss\n",
        "        labels: Int[torch.Tensor, \"entries\"] = torch.arange(logits_per_image.shape[0])\n",
        "\n",
        "        return torch.stack((\n",
        "            nn.functional.cross_entropy(logits_per_image, labels),\n",
        "            nn.functional.cross_entropy(logits_per_text, labels),\n",
        "        )).mean()"
      ],
      "metadata": {
        "id": "WVxnFRdLffaW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "contrastive_loss_fn: t.Callable[\n",
        "    [\n",
        "        Float[torch.Tensor, \"entries 1024\"],\n",
        "        Float[torch.Tensor, \"entries 1024\"],\n",
        "        Float[torch.Tensor, \"1\"],\n",
        "    ],\n",
        "    Float[torch.Tensor, \"1\"],\n",
        "] = ClipLoss()"
      ],
      "metadata": {
        "id": "wLrh0RM4fh6t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def contrastive_training_step(\n",
        "    model: ClipFlyp,\n",
        "    data_loader: DataLoader[tuple[TensorImage, list[str]]],\n",
        "    optimizer: torch.optim.Optimizer,\n",
        ") -> float:\n",
        "    running_loss: float = 0.0\n",
        "    progress = tqdm(data_loader, desc=\"training\")\n",
        "\n",
        "    model.train()\n",
        "\n",
        "    entries: tuple[tuple[TensorImage, list[str]], ...]\n",
        "    entry: tuple[TensorImage, list[str]]\n",
        "\n",
        "    for iter, entries in zip(it.count(1), progress):\n",
        "        # forward computation\n",
        "        imgs_features, txts_features, logit_scale = model(\n",
        "            [(img, prompts[0]) for img, prompts in entries]\n",
        "        )\n",
        "\n",
        "        # calculate loss\n",
        "        loss: Float[torch.Tensor, \"1\"] = contrastive_loss_fn(imgs_features, txts_features, logit_scale)\n",
        "        running_loss += loss.item()\n",
        "\n",
        "        # optimizer zero grad\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # loss backward\n",
        "        loss.backward()\n",
        "\n",
        "        # optimizer step\n",
        "        optimizer.step()\n",
        "\n",
        "        # Note: we clamp to 4.6052 = ln(100), as in the original paper.\n",
        "        with torch.no_grad():\n",
        "            model.core.logit_scale.clamp_(0, math.log(100))\n",
        "\n",
        "            progress.set_postfix({\"loss\": running_loss / iter}, refresh=False)\n",
        "\n",
        "    return running_loss / len(data_loader)"
      ],
      "metadata": {
        "id": "DJdQ6ZsjfjfB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def contrastive_test_step(\n",
        "    model: ClipFlyp,\n",
        "    data_loader: DataLoader[tuple[TensorImage, list[str]]],\n",
        ") -> float:\n",
        "    running_loss: float = 0.0\n",
        "    progress = tqdm(data_loader, desc=\"testing\")\n",
        "\n",
        "    model.eval()\n",
        "\n",
        "    with torch.inference_mode():\n",
        "        entries: tuple[tuple[TensorImage, list[str]], ...]\n",
        "        entry: tuple[TensorImage, list[str]]\n",
        "\n",
        "        for iter, entries in zip(it.count(1), progress):\n",
        "            # forward computation\n",
        "            imgs_features, txts_features, logit_scale = model(\n",
        "                [(img, prompts[0]) for img, prompts in entries]\n",
        "            )\n",
        "\n",
        "            # calculate loss\n",
        "            loss: Float[torch.Tensor, \"1\"] = contrastive_loss_fn(imgs_features, txts_features, logit_scale)\n",
        "            running_loss += loss.item()\n",
        "\n",
        "            progress.set_postfix({\"loss\": running_loss / iter}, refresh=False)\n",
        "\n",
        "    return running_loss / len(data_loader)"
      ],
      "metadata": {
        "id": "BwHmnZ8zgkxV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def contrastive_showtime(\n",
        "    model: ClipFlyp,\n",
        "    spli2loader: dict[Split, DataLoader[tuple[TensorImage, list[str]]]],\n",
        "    writer: SummaryWriter,\n",
        "    global_step: int,\n",
        ") -> None:\n",
        "    model.eval()\n",
        "\n",
        "    with torch.inference_mode():\n",
        "\n",
        "        for split, data_loader in spli2loader.items():\n",
        "\n",
        "            progress = tqdm(data_loader, desc=\"showtime [{split}]\")\n",
        "\n",
        "            entries: tuple[tuple[TensorImage, list[str]], ...]\n",
        "            entry: tuple[TensorImage, list[str]]\n",
        "\n",
        "            for iter, entries in zip(it.count(1), progress):\n",
        "\n",
        "                # forward computation\n",
        "                imgs_features, txts_features, _ = model([\n",
        "                    (img, prompts[0])\n",
        "                    for img, prompts in entries\n",
        "                ])\n",
        "\n",
        "                imgs_features: Float[torch.Tensor, \"entries 1024\"] = imgs_features / imgs_features.norm(dim=-1, keepdim=True)\n",
        "                txts_features: Float[torch.Tensor, \"entries 1024\"] = txts_features / txts_features.norm(dim=-1, keepdim=True)\n",
        "                similarity: Float[torch.Tensor, \"entries entries\"] = (txts_features @ imgs_features.T).cpu()\n",
        "\n",
        "                f: plt.Figure\n",
        "                ax: plt.Axes\n",
        "                f, ax = plt.subplots(1, 1, figsize=(10, 8))\n",
        "\n",
        "                ax.imshow(similarity, vmin=torch.min(similarity).item(), vmax=torch.max(similarity).item())\n",
        "\n",
        "                ax.set_yticks(\n",
        "                    range(len(entries)),\n",
        "                    [\"\\n\".join(prompts) for _, prompts in entries],\n",
        "                    fontsize=10,\n",
        "                )\n",
        "                ax.set_xticks([])\n",
        "\n",
        "                for i, image in enumerate([ crop for crop, _ in entries ]):\n",
        "                    ax.imshow(\n",
        "                        image.permute(1, 2, 0).cpu(),\n",
        "                        extent=(i - 0.5, i + 0.5, -1.6, -0.6),\n",
        "                        origin=\"lower\",\n",
        "                    )\n",
        "\n",
        "                for x in range(similarity.shape[1]):\n",
        "                    for y in range(similarity.shape[0]):\n",
        "                        ax.text(\n",
        "                            x,\n",
        "                            y,\n",
        "                            f\"{similarity[y, x]:.2f}\",\n",
        "                            ha=\"center\",\n",
        "                            va=\"center\",\n",
        "                            size=12,\n",
        "                        )\n",
        "\n",
        "                for side in [\"left\", \"top\", \"right\", \"bottom\"]:\n",
        "                    f.gca().spines[side].set_visible(False)\n",
        "\n",
        "                ax.set_xlim([-0.5, len(entries) - 0.5])\n",
        "                ax.set_ylim([len(entries) + 0.5, -2])\n",
        "\n",
        "                f.tight_layout()\n",
        "\n",
        "                writer.add_figure(tag=f\"matrix {iter}/{split}\", figure=f, global_step=global_step)"
      ],
      "metadata": {
        "id": "cnTlJB12gmSL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qqM4rteUp_mP"
      },
      "source": [
        "## 12 Exploiting self-attention to provide contextualized latent space representations\n",
        "\n",
        "`CLIP context`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OH9w1Oy5_ZXZ"
      },
      "source": [
        "As demonstrated by the numbers achieved by our fine-tuned architecture on the test set of the RefCOCOg dataset, our trained model is very well prepared in mapping the visual appearance of the image regions delimited by the proposed bounding boxes together with the semantic of the given textual description. Even though the algorithm achieves remarkable results, by manually inspecting the wrong predictions we have noticed that the AI agent makes mistakes in presence of location words such as \"in the middle\", \"on the right\", \"on the left\" and similar. However, if we carefully reason about the pipeline pursued by our software to make the final prediction (Figure 13) we understand that the described deficiency is justifiable. Intuitively, with our implementation, the model can merely look at each image region proposed in step 1 in isolation, disregarding the surrounding elements which populates the field of view. Under such circumstances even a human would be in trouble in understanding which is \"The dog on the left\" and which is \"The dog on the right\" (Figure 12). In order to figure out which is the dog referred to by the natural language description, we need to grasp the content of the picture as a whole. Even though achieving a general understanding of the global scene is easily solvable in less than one second by a human observer, the task is not trivial for a machine. The purpose of this section is to describe our solution proposal to partially overcome this limitation.\n",
        "\n",
        "The crucial observation behind the idea that we have come up with in order to accomplish this, is that we need a way to enhance the latent representations of visual and textual prompts so that the obtained encodings embody the overall context of the given picture. Intuitively, the image crops and the textual description have to be interpreted as words in an arbitrary long sentence. The encoding of a bounding box region, just as a pronoun, might acquire different meanings depending on the rest of the portrayed phrase. Consistently with this perspective, we have devised a self-attention mechanism as originally proposed by the famous \"Attention Is All You Need\" research paper [24]. For the sake of clarity the architecture is summarized in Figure 34. In this additional proposed encoding procedure, the visual and textual CLIP embeddings are fed as query, key and value inputs of a self-attention module. By doing so, the algorithm still has severe problems in realizing how the objects are reciprocally disseminated in the environment. However, a cropped tasty apple within a bounding box is no longer analyzed in isolation, but its feature representation incorporates the presence of the other beautiful fruits in the basket (Figure 35). Given the query: \u201cA yellow banana fruit in a basket\u201d $p$, the probability assigned to the apple bounding box $b$ is probably still not negligible. However, this time, the cosine similarity between $p$ and $b$ is mitigated by the awareness that somewhere in the picture there is also a bunch of bananas.\n",
        "\n",
        "As always, before launching the training procedure on our powerful but time limited Azure GPUs, we have executed some representative experiments on Google Collaboratory. Sadly, the displayed results are not promising at all. After several unsuccessful attempts, we believe that the reason beyond these poor feedbacks is related to the limited portion of the dataset that we are inevitably constrained to consider by the limited resources made available by Google Collaboratory. Indeed, when dealing with attention modules, very large training data are typically mandatory in order to appreciate satisfactory outcomes. Unfortunately, at this point of the project, we had spent most of our GPU execution time on Azure and we considered it too hazardous to spend the remaining hours of execution for something that could potentially result in a discouraging stalemate. We strongly believe that training this architecture on the whole dataset for at least 50 epochs is a worthy future direction that we cannot afford to consider in this work. At the end of the day, given that the deep neural network trained with contrastive learning have shown wonderful achievements, we consider it more profitable and interesting to investigate promising strategies to further improve the results obtained so far. In this regard, we discuss our proposed solutions to enhance the generalization capabilities of our model in the next section.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ONB_SfvhGn3g"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/34.png)\n",
        "> \n",
        "> **Figure 34**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOUsjNFeaIAA"
      },
      "source": [
        "> ![fruits.jpg](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/35.jpeg)\n",
        "> \n",
        "> **Figure 35**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 12.1 Code"
      ],
      "metadata": {
        "id": "2fGzVoc2gxFh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipContexCore(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        img_encoder: nn.Module,  # visual encoder\n",
        "        txt_encoder: nn.Module,  # natural language prompts encoder\n",
        "    ):\n",
        "        super().__init__()\n",
        "        self.img_encoder = img_encoder\n",
        "        self.txt_encoder = txt_encoder\n",
        "        self.attention = nn.MultiheadAttention(embed_dim=1024, num_heads=1)\n",
        "\n",
        "    def contextualize(\n",
        "        self,\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"],\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"],\n",
        "    ) -> tuple[Float[torch.Tensor, \"crops 1024\"], Float[torch.Tensor, \"prompts 1024\"]]:\n",
        "        # concatenate image embeedings and prompt embeedings in the same latent context\n",
        "        concat: Float[torch.Tensor, \"crops+prompts 1024\"] = torch.cat((crops_z, prompts_z), dim=0)\n",
        "\n",
        "        contextualized: Float[torch.Tensor, \"crops+prompts 1024\"]\n",
        "        contextualized, _ = self.attention(concat, concat, concat)\n",
        "\n",
        "        # retrive image_features and text_features by means of the previously stored indexes\n",
        "        return contextualized[: crops_z.shape[0]], contextualized[-prompts_z.shape[0] :]\n",
        "\n",
        "    def cosine_similarity(\n",
        "        self,\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"],\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"],\n",
        "    ) -> Float[torch.Tensor, \"prompts crops\"]:\n",
        "        # normalise the image and the text\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"] = crops_z / crops_z.norm(dim=-1, keepdim=True)\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"] = prompts_z / prompts_z.norm(dim=-1, keepdim=True)\n",
        "\n",
        "        # evaluate the cosine similarity between the sets of features\n",
        "        return prompts_z @ crops_z.T\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        crops: Float[torch.Tensor, \"crops 3 244 244\"],\n",
        "        prompts: Int[torch.Tensor, \"prompts 77\"],\n",
        "    ) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        # step 1: compute crop representation in the latent space\n",
        "        crops_z: Float[torch.Tensor, \"crops 1024\"] = self.img_encoder(crops)\n",
        "\n",
        "        # step 2: compute prompt representation in the latent space\n",
        "        prompts_z: Float[torch.Tensor, \"prompts 1024\"] = self.txt_encoder(prompts)\n",
        "\n",
        "        # step 3: refine the latent representation of each text and image according to the overall context by means of the attention mechanism\n",
        "        crop_context_z: Float[torch.Tensor, \"crops 1024\"]\n",
        "        prompt_context_z: Float[torch.Tensor, \"prompts 1024\"]\n",
        "        crop_context_z, prompt_context_z = self.contextualize(crops_z, prompts_z)\n",
        "\n",
        "        # step 4: evaluate logits\n",
        "        similarity_matrix: Float[torch.Tensor, \"prompts crops\"] = self.cosine_similarity(crop_context_z, prompt_context_z)\n",
        "\n",
        "        # step 5: crops classification\n",
        "        return torch.mean(similarity_matrix, dim=0)"
      ],
      "metadata": {
        "id": "1J5mEAuOgvoj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class ClipContex(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        img_encoder: nn.Module,  # visual encoder\n",
        "        txt_encoder: nn.Module,  # natural language prompts encoder\n",
        "    ):\n",
        "        super().__init__()\n",
        "        self.core = ClipContexCore(img_encoder, txt_encoder)\n",
        "\n",
        "        self.img_preprocess: Compose = preprocess\n",
        "        self.txt_preprocess: t.Callable[[t.Union[str, list[str]]], Float[torch.Tensor, \"77\"]] = clip.tokenize\n",
        "\n",
        "    def forward(\n",
        "        self, crops: list[TensorImage], prompts: list[str]\n",
        "    ) -> Float[torch.Tensor, \"crops 1\"]:\n",
        "        # step 1: preprocess crops as required by the visual encoder\n",
        "        with torch.no_grad():\n",
        "            crops_preprocessed: Float[torch.Tensor, \"crops 3 244 244\"] = torch.stack([\n",
        "                self.img_preprocess(crop)\n",
        "                for crop in crops\n",
        "            ])\n",
        "\n",
        "        # step 2: preprocess prompts as required by the text encoder\n",
        "        with torch.no_grad():\n",
        "            prompts_preprocessed: Int[torch.Tensor, \"prompts 77\"] = self.txt_preprocess(\n",
        "                prompts\n",
        "            )\n",
        "\n",
        "        return self.core(crops_preprocessed, prompts_preprocessed)"
      ],
      "metadata": {
        "id": "EYdiypAPg18M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eval_summary(\n",
        "    ClipContex(\n",
        "        img_encoder=clip_frozen_img_encoder,\n",
        "        txt_encoder=clip_frozen_txt_encoder,\n",
        "    ).to(device).core\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6hYWzbAog3lN",
        "outputId": "802cd4ae-2493-414f-ebc4-c2c1cf5b8724"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "============================================================================================================================================\n",
              "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Trainable\n",
              "============================================================================================================================================\n",
              "ClipContexCore                           [5, 3, 244, 244]          [5]                       --                        True\n",
              "\u251c\u2500ClipFrozenImgEnc: 1-1                  [5, 3, 244, 244]          [5, 1024]                 --                        --\n",
              "\u251c\u2500ClipFrozenTxtEnc: 1-2                  [2, 77]                   [2, 1024]                 --                        --\n",
              "\u251c\u2500MultiheadAttention: 1-3                [7, 1024]                 [7, 1024]                 4,198,400                 True\n",
              "============================================================================================================================================\n",
              "Total params: 4,198,400\n",
              "Trainable params: 4,198,400\n",
              "Non-trainable params: 0\n",
              "Total mult-adds (M): 0\n",
              "============================================================================================================================================\n",
              "Input size (MB): 3.57\n",
              "Forward/backward pass size (MB): 0.00\n",
              "Params size (MB): 0.00\n",
              "Estimated Total Size (MB): 3.57\n",
              "============================================================================================================================================"
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0QeU8Y0BrAor"
      },
      "source": [
        "## 13 Strategies to improve model generalization"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The results obtained with the architecture fine-tuned in a contrastive learning fashion are outstanding and outperform all the other deep neural networks we have proposed so far. Nevertheless, there is still space for improvement. The purpose of this section is to present and provide an open source implementation of the solutions that we have put in place in order to refine our most promising architecture with the aim of further improving its generalization capabilities on the downstream task at hand."
      ],
      "metadata": {
        "id": "3_c_tmzgIBl7"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6hwK759osIkO"
      },
      "source": [
        "### 13.1 Hyperparameter tuning\n",
        "\n",
        "`FLYP optuna`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the architectures presented so far we have adopted the hyperparameter values which are commonly used in deep learning codes. Although the proposed configuration led to satisfactory results, with the aim of exploring more effectively the highly non-linear landscape of our complex objective function, we have leveraged on automatic optimization tools for hyperparameter tuning. Specifically, in the implementation presented in the following of this section, we use [Optuna](https://optuna.org/), an automatic hyperparameter optimization software framework, particularly designed for machine learning. The software enables efficient hyperparameter optimization by adopting state-of-the-art algorithms for sampling hyperparameters and pruning efficiently unpromising trials. We execute Optuna with the combination of sampling and pruning algorithms suggested by the official documentation for our use case scenario. More in depth, the sampling algorithm of choice is Tree-structured Parzen Estimator [25] while the pruning algorithm is Hyperband [26].\n",
        "Unfortunately, the computational complexity of the search through the hyperparameter space rapidly increases with the increase of the number of hyperparameters to be optimized. Because of our limited computational resources, we can not afford to finetune the values of all the hyperparameters which populate our architecture. As a consequence, we have selected the variables which in our opinion have the largest impact on the training procedure.\n",
        "\n",
        "\n",
        "*   optimizer: undoubtedly, the model's convergence and performance highly depend on the adopted optimizer. For this hyperparameter optimization process we select two widely used algorithms: Stochastic Gradient Descent (SGD) and Adam.\n",
        "*   learning rate: the learning rate determines the step size at each iteration while moving toward a desirable basin of attraction. Certainly, the pace of the solution space explorer significantly impacts the final outcome. In our implementation Optuna is asked to find the optimal learning rate parameter within the interval [0.00001, 0.1].\n",
        "* dropout: in Section 11.1 we write about the regularization techniques that we have put in place in order to prevent overfitting. Among these, the dropout rate has turned out to be one of the most influential. Therefore, in the implementation proposed in the following of this section, we configure Optuna to search for the most suitable dropout rate value within the interval [0.1, 0.75].\n",
        "\n",
        "We let Optuna play with these parameters for five hours on Azure GPUs. Obviously, a longer period of time would have been more desirable but we have been forced to select an interval compatible with the resources we had at that point of the project.\n",
        "\n",
        "The following (Figure 37) are the hyperparameters suggested by Optuna at the end of its execution."
      ],
      "metadata": {
        "id": "xwAunTLtIFAa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Given this optimized hyperparameter configuration we have assessed its goodness repeating the overall training procedure with the recommended parameters. Factually, a more accurate selection of the hyperparameters which govern the behavior of the training, led to better achievements as reported below."
      ],
      "metadata": {
        "id": "HRWli9lHIk8S"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nTq48C5o7nwE"
      },
      "source": [
        "> ![Screenshot 2023-09-01 at 01.24.14.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/36.png)\n",
        "> \n",
        "> **Figure 36**\n",
        "> \n",
        "> `FLYP optuna` loss"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "OPTUNA_BATCH_SIZE: int = 1024\n",
        "OPTUNA_LIMIT: int = 30 * OPTUNA_BATCH_SIZE\n",
        "OPTUNA_EPOCHS: int = 10"
      ],
      "metadata": {
        "id": "nq0caI2XiyJJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "optuna_split2loader: dict[Split, DataLoader[tuple[TensorImage, list[str]]]] = {\n",
        "    split: DataLoader(\n",
        "        dataset=Coco4ContrastiveDataset(split=split, limit=OPTUNA_LIMIT),\n",
        "        generator=g,\n",
        "        batch_size=OPTUNA_BATCH_SIZE,\n",
        "        collate_fn=lambda x: x,\n",
        "        shuffle=(split == \"train\"),\n",
        "    )\n",
        "    for split in [\"train\", \"val\"]\n",
        "}"
      ],
      "metadata": {
        "id": "Ec97ws-9izmJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def objective(trial: Trial):\n",
        "\n",
        "    lr: float = trial.suggest_float(\"learning rate\", 1e-5, .1, log=True)\n",
        "    p: float = trial.suggest_float(\"dropout\", .1, .9)\n",
        "    optim: t.Literal[\"Adam\", \"SGD\"] = trial.suggest_categorical(\"optimizer\", [\"Adam\", \"SGD\"])\n",
        "\n",
        "    optuna_model: ClipFlyp = ClipFlyp(p=p).to(device)\n",
        "\n",
        "    match optim:\n",
        "      case \"Adam\":\n",
        "            optimizer: torch.optim.Optimizer = torch.optim.Adam(\n",
        "                params=optuna_model.parameters(),\n",
        "                lr=lr,\n",
        "                weight_decay=.01\n",
        "            )\n",
        "\n",
        "      case \"SGD\":\n",
        "            optimizer: torch.optim.Optimizer = torch.optim.SGD(\n",
        "                params=optuna_model.parameters(),\n",
        "                lr=lr,\n",
        "                weight_decay=.01,\n",
        "                momentum=.9\n",
        "            )\n",
        "\n",
        "    for epoch in trange(OPTUNA_EPOCHS):\n",
        "\n",
        "        contrastive_training_step(\n",
        "            model = optuna_model,\n",
        "            data_loader = optuna_split2loader[\"train\"],\n",
        "            optimizer = optimizer,\n",
        "        )\n",
        "\n",
        "        val_loss = contrastive_test_step(\n",
        "            model = optuna_model,\n",
        "            data_loader = optuna_split2loader[\"val\"],\n",
        "        )\n",
        "\n",
        "        trial.report(val_loss, epoch)\n",
        "\n",
        "        # Handle pruning based on the intermediate value.\n",
        "        if trial.should_prune():\n",
        "            raise optuna.TrialPruned()\n",
        "\n",
        "    return val_loss"
      ],
      "metadata": {
        "id": "dW67yjZWi1EF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
        "\n",
        "study: optuna.study.Study = optuna.create_study(\n",
        "    study_name=\"optuna-hyperparameter-optimization\",\n",
        "    direction=\"minimize\",\n",
        "    pruner=optuna.pruners.HyperbandPruner(),\n",
        "    load_if_exists=False,\n",
        ")\n",
        "\n",
        "# study.optimize(\n",
        "#     func=objective,\n",
        "#     timeout=5 * 60 * 60,\n",
        "#     show_progress_bar=True,\n",
        "# )"
      ],
      "metadata": {
        "id": "S1UTyZ17jBIS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "D0ch-VD1ixTL"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PCP_tUySrR66"
      },
      "outputs": [],
      "source": [
        "study: Study = optuna.create_study(\n",
        "    storage=RDBStorage(\"sqlite:///assets/optuna.db\"),\n",
        "    study_name=\"flyp\",\n",
        "    load_if_exists=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qbKR3w0qrgHX"
      },
      "outputs": [],
      "source": [
        "plot_parallel_coordinate(study)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3xfjRUzhfan"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/37.png)\n",
        "> \n",
        "> **Figure 37**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-LWstZJ0r0-a"
      },
      "outputs": [],
      "source": [
        "plot_param_importances(study)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rfceFJQOhahS"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/38.png)\n",
        "> \n",
        "> **Figure 38**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X3XHg4Iur2sL"
      },
      "outputs": [],
      "source": [
        "plot_timeline(study)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BK8zp2mYhQn8"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/39.png)\n",
        "> \n",
        "> **Figure 39**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3I2pqJ1sLdO"
      },
      "source": [
        "### 13.2 Data augmentation and noise injection\n",
        "\n",
        "`FLYP augmented`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As detailed in Section 5, for each annotation the RefCOCOg dataset provides one, two or three equivalent descriptions. Up to this point, we have always considered only one of them. Without doubt, basing the decisions on all the available referring expressions rather than a subset of them would be more desirable. It is important to recall that the training loop of our contrastive learning procedure is extremely efficient and it is able to process the entire dataset in an acceptable amount of time. With this in mind, with the next pieces of code, not only we are able to consider all the prompts in the dataset, but we also succeed in developing data augmentation and noise injection techniques with the purpose of enhancing the generalization capabilities of our model."
      ],
      "metadata": {
        "id": "MDmi3yQIJppS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 13.2.1 Text augmentation"
      ],
      "metadata": {
        "id": "nhqImNYCJVDi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Given a natural language description referring to a region of interest of the input image, our aim is to produce a set of different, but semantically equivalent, expressions. To this end, we have developed a set of possible transformations which are randomly applied on the input sequences in order to produce a desired number of phrases with the same meaning. The purpose of the paragraphs of this subsection is to outline the text augmentation techniques that we have applied in this work.\n",
        "\n",
        "Inspired by the paper of CLIP [2], we generate new sentences following the concept of prompt engineering [27] [28], which simply consist in arbitrary appending a set of suggested prefixes (\"A photo of {}\", \"A picture of {}\", \"An image of {}\", \"This is {}\", \"We can see {}\") at the beginning of a given input sequence.\n",
        "\n",
        "In addition to this, we also leverage on the impressive skills of more sophisticated modern sequence-to-sequence text generation models which are able, given a sentence, to generate a bunch of synonyms with the same meaning [29]. More in depth, we have succeeded in configuring and executing the following powerful NLP algorithms:\n",
        "\n",
        "\n",
        "*   EDA [30]. The Easy Data Augmentation algorithm consists of four simple operations: synonym replacement, random insertion, random swap, and random deletion. For our scopes, we have used only the synonym replacement component of the model. The reference guidelines for using EDA are clearly documented [at this GitHub repository](https://github.com/dsfsi/textaugment).\n",
        "*   PEGASUS [31]. We use a fine-tuned version specifically designed for paraphrasing of the PEGASUS transformer-based encoder-decoder model. The reference implementation of this deep neural network is provided by [huggingface.co at this link](https://huggingface.co/tuner007/pegasus_paraphrase)\n",
        "* BART [32]. Additionally, we have also been able to employ a large BART sequence-to-sequence generation model fine-tuned on three paraphrase datasets made available by [huggingface.co at this URL address](https://huggingface.co/eugenesiow/bart-paraphrase)\n"
      ],
      "metadata": {
        "id": "26lfjlvAJrgP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "> ```\n",
        "> a apple desktop computer\n",
        "> - a apple desktop calculator\n",
        "> \n",
        "> a blonde woman in a white shirt and long black skirt\n",
        "> - We can see a blonde woman in a white shirt and long black skirt\n",
        "> \n",
        "> an old truck covered in snow except for the grill and door\n",
        "> - An old truck is covered in snow.\n",
        "> \n",
        "> the adult giraffe\n",
        "> - the adult camelopard\n",
        "> ```\n",
        "> \n",
        "> **Figure 40**"
      ],
      "metadata": {
        "id": "BisMIX5ny97P"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 13.2.2 Image augmentation"
      ],
      "metadata": {
        "id": "eq6wXIXzJZDo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Fascinatingly, in every human language, given a sentence describing a specific object in a scene or a certain situation, there exist endless equivalent expressions characterized by the same meaning. \"Sara exhibited considerable bravery during the challenging ordeal.\"; \"Sara displayed significant courage throughout the demanding experience\"; \"Sare showcased noteworthy valor while facing the arduous trial.\". Each of these clauses can be used interchangeably without affecting our semantic comprehension. Equivalently, given a portion of an image delimited by a rectangular bounding box, we can think about endless possible modifications of its appearance that do not affect our understanding of the content. For instance, we can straightforwardly recognize a cat in a crop even if it is gray scaled, rotated or slightly blurred. Following this idea, given a picture and a sequence of semantically equivalent expressions referring to a particular object in the field of view, for each description we augment the dataset introducing a \"visual synonym\" of the bounding box region which delimits the object of interest. To accomplish this, we apply a visual aberration randomly chosen from a set of imported [torchvision transformations](https://pytorch.org/vision/stable/transforms.html).\n"
      ],
      "metadata": {
        "id": "CEGKeWqlMKEk"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8qpqGbGSxczV"
      },
      "source": [
        "> ![color.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/40.png)\n",
        "> \n",
        "> **Figure 41**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 13.2.3 Noise injection"
      ],
      "metadata": {
        "id": "jp__NdlJJaPL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Experimental results suggest that noise injection might improve the generalization ability of the resulting neural network\n",
        "[33]. In line with this widely used concept, we enrich our text and visual data augmentation solution with slight perturbations applied randomly.\n",
        "\n",
        "As regards textual data, we have experimentally noticed that the aforementioned EDA generation algorithm sometimes fails to replace text words with appropriate synonyms. In these cases, the resulting sentences do not precisely reflect the content of the initial phrase. The frequency at which this occurs makes EDA not only a useful tool for data augmentation but unintentionally also a valuable noise injection mechanism.\n",
        "\n",
        "On the other hand, the situation is considerably easier for images due to the vast amount of existing computer vision libraries. In our work we perturb the visual content of a random subset of image crops of our dataset by blurring the content of the pixels."
      ],
      "metadata": {
        "id": "mzDfoL8ZMa34"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5uVrvD7ZxQfu"
      },
      "source": [
        "> ![noise.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/41.png)\n",
        "> \n",
        "> **Figure 42**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 13.3 Code"
      ],
      "metadata": {
        "id": "SgrwIc-ihSKE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def contrastive_training_step_with_synonyms(\n",
        "    model: ClipFlyp,\n",
        "    data_loader: DataLoader[tuple[TensorImage, list[str]]],\n",
        "    synonyms: int,\n",
        "    optimizer: torch.optim.Optimizer,\n",
        ") -> float:\n",
        "    running_loss: float = 0.0\n",
        "    progress = tqdm(data_loader, desc=\"training\")\n",
        "\n",
        "    model.train()\n",
        "\n",
        "    entries: list[list[tuple[TensorImage, str]]]\n",
        "    entry: list[tuple[TensorImage, str]]\n",
        "\n",
        "    for iter, entries in zip(it.count(1), progress):\n",
        "\n",
        "        # forward computation\n",
        "        imgs_features: Float[torch.Tensor, \"entries*synonyms 1024\"]\n",
        "        txts_features: Float[torch.Tensor, \"entries*synonyms 1024\"]\n",
        "        logit_scale: Float[torch.Tensor, \"1\"]\n",
        "        imgs_features, txts_features, logit_scale = model(list(it.chain(*entries)))\n",
        "\n",
        "        imgs_features_3d: Float[torch.Tensor, \"entries synonyms 1024\"] = imgs_features.view(len(entries), synonyms, 1024)\n",
        "        imgs_features_3d: Float[torch.Tensor, \"synonyms entries 1024\"] = imgs_features_3d.transpose(0, 1)\n",
        "\n",
        "        txts_features_3d: Float[torch.Tensor, \"entries synonyms 1024\"] = txts_features.view(len(entries), synonyms, 1024)\n",
        "        txts_features_3d: Float[torch.Tensor, \"synonyms entries 1024\"] = txts_features_3d.transpose(0, 1)\n",
        "\n",
        "        # calculate loss\n",
        "        loss: Float[torch.Tensor, \"1\"] = torch.stack([\n",
        "            loss_fn(imgs_features_2d, txts_features_2d, logit_scale)\n",
        "            for imgs_features_2d, txts_features_2d in zip(imgs_features_3d, txts_features_3d)\n",
        "        ]).mean()\n",
        "        running_loss += loss.item()\n",
        "\n",
        "        # optimizer zero grad\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # loss backward\n",
        "        loss.backward()\n",
        "\n",
        "        # optimizer step\n",
        "        optimizer.step()\n",
        "\n",
        "        # Note: we clamp to 4.6052 = ln(100), as in the original paper.\n",
        "        with torch.no_grad():\n",
        "            model.core.logit_scale.clamp_(0, math.log(100))\n",
        "\n",
        "            progress.set_postfix({\"loss\": running_loss / iter}, refresh=False)\n",
        "\n",
        "    return running_loss / len(data_loader)"
      ],
      "metadata": {
        "id": "sDWGQ7WkhTSy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "2eW1X_Jchjku"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# EDA\n",
        "# paper: https://aclanthology.org/D19-1670.pdf\n",
        "# paper: https://arxiv.org/abs/1907.03752\n",
        "# code reference: https://github.com/dsfsi/textaugment\n",
        "from textaugment import EDA\n",
        "\n",
        "import nltk  # NLTK is a leading platform for building Python programs to work with human language data\n",
        "\n",
        "nltk.download(\"stopwords\")\n",
        "nltk.download(\"wordnet\")"
      ],
      "metadata": {
        "id": "G5kYRC5vhVTg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# PEGASUS fine-tuned for paraphrasing\n",
        "# paper: https://arxiv.org/abs/1912.08777\n",
        "# code reference: https://huggingface.co/tuner007/pegasus_paraphrase\n",
        "from transformers import PegasusForConditionalGeneration, PegasusTokenizer\n",
        "\n",
        "pegasus_model_name = \"tuner007/pegasus_paraphrase\"\n",
        "pegasus_tokenizer = PegasusTokenizer.from_pretrained(pegasus_model_name)\n",
        "pegasus_model = PegasusForConditionalGeneration.from_pretrained(pegasus_model_name).to(device)\n",
        "\n",
        "pegasus_model.eval()\n",
        "\n",
        "for p in pegasus_model.parameters():\n",
        "    p.requires_grad = False"
      ],
      "metadata": {
        "id": "lRCNK11rhlsj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# A large BART seq2seq (text2text generation) model fine-tuned on 3 paraphrase datasets.\n",
        "# paper: https://arxiv.org/abs/1910.13461\n",
        "# code reference: https://huggingface.co/eugenesiow/bart-paraphrase\n",
        "from transformers import BartForConditionalGeneration, BartTokenizer\n",
        "\n",
        "bart_model_name = \"eugenesiow/bart-paraphrase\"\n",
        "bart_tokenizer = BartTokenizer.from_pretrained(bart_model_name)\n",
        "bart_model = BartForConditionalGeneration.from_pretrained(bart_model_name).to(device)\n",
        "\n",
        "bart_model.eval()\n",
        "\n",
        "for p in bart_model.parameters():\n",
        "    p.requires_grad = False"
      ],
      "metadata": {
        "id": "VIMAkZdbhnVz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def pegasus(txt: str) -> str:\n",
        "    with torch.inference_mode():\n",
        "        batch: dict[str, Int[torch.Tensor, \"1 P\"]] = pegasus_tokenizer(\n",
        "            [txt],\n",
        "            truncation=True,\n",
        "            padding=\"longest\",\n",
        "            max_length=60,\n",
        "            return_tensors=\"pt\",\n",
        "        )\n",
        "        translated: Int[torch.Tensor, \"1 X\"] = pegasus_model.generate(\n",
        "            **batch,\n",
        "            max_length=60,\n",
        "            num_beams=10,\n",
        "            num_return_sequences=1,\n",
        "            temperature=1.5\n",
        "        )\n",
        "        [out] = pegasus_tokenizer.batch_decode(translated, skip_special_tokens=True)\n",
        "        return out"
      ],
      "metadata": {
        "id": "oGsh-OAuho7k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def bart(txt: str) -> str:\n",
        "    with torch.inference_mode():\n",
        "        batch: dict[str, Int[torch.Tensor, \"1 P\"]] = bart_tokenizer(txt, return_tensors=\"pt\")\n",
        "        translated: Int[torch.Tensor, \"1 X\"] = bart_model.generate(batch[\"input_ids\"])\n",
        "        [out] = bart_tokenizer.batch_decode(translated, skip_special_tokens=True)\n",
        "        return out"
      ],
      "metadata": {
        "id": "2Hfd-L1ThqXj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "eda: EDA = EDA(random_state=42)"
      ],
      "metadata": {
        "id": "nUTNA720hrph"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "txt_transform: RandomChoice = RandomChoice([\n",
        "    \"A photo of {}\".format,\n",
        "    \"A picture of {}\".format,\n",
        "    \"An image of {}\".format,\n",
        "    \"This is {}\".format,\n",
        "    \"We can see {}\".format,\n",
        "    eda.synonym_replacement,\n",
        "    pegasus,\n",
        "    bart,\n",
        "])"
      ],
      "metadata": {
        "id": "-Rgl_VQhhs8J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "img_transform: RandomChoice = RandomChoice([\n",
        "    ColorJitter(brightness=0.5, hue=0.3), # randomly changes the brightness, saturation, and other properties of an image\n",
        "    GaussianBlur(kernel_size=(5, 9), sigma=(0.1, 5)),  # performs gaussian blur transform on an image\n",
        "    RandomPosterize(bits=2),  # randomly posterizes the image by reducing the number of bits of each color channel\n",
        "    RandomSolarize(threshold=192.0),  # randomly solarizes the image by inverting all pixel values above the threshold\n",
        "    RandomAdjustSharpness(sharpness_factor=2),  # randomly adjusts the sharpness of the given image\n",
        "    RandomAutocontrast(),  # randomly applies autocontrast to the given image\n",
        "    RandomEqualize(),  # randomly equalizes the histogram of the given image\n",
        "    Grayscale(num_output_channels=3),  # converts an image to grayscale\n",
        "])"
      ],
      "metadata": {
        "id": "bLIRfbr_huNC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "SYNONYMS: int = 2"
      ],
      "metadata": {
        "id": "cbzhhGNHiBLl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def augment(batch: list[tuple[TensorImage, list[str]]]) -> list[list[tuple[TensorImage, str]]]:\n",
        "    return [\n",
        "        list(zip(\n",
        "            random.sample(\n",
        "                [img] +\n",
        "                [img_transform(img) for _ in range(SYNONYMS - 1)],\n",
        "                SYNONYMS\n",
        "            ),\n",
        "            random.sample(\n",
        "                prompts +\n",
        "                [txt_transform(random.choice(prompts)) for _ in range(SYNONYMS - len(prompts))],\n",
        "                SYNONYMS\n",
        "            )\n",
        "        ))\n",
        "        for img, prompts in batch\n",
        "    ]"
      ],
      "metadata": {
        "id": "2SI_HhPIiCfE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 13.4 Results"
      ],
      "metadata": {
        "id": "Ma8C6dGhJgtZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the following part of the notebook we present our implementation proposal which incorporates the data augmentation logics presented in this section. Consistently with the aforementioned concepts, in this novel implementation, during training our batch is no more populated by $B$ (bounding box image, description) pairs. Now, given a configurable number of synonyms $m$, the resulting batch has the following structure:"
      ],
      "metadata": {
        "id": "PzL9iGKMM2Hj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "$$\n",
        "\\begin{bmatrix}\n",
        "\\text{bbox}_{1, 1} & \\text{bbox}_{1, 2} & \\cdots & \\text{bbox}_{1, m} \\\\\n",
        "\\text{bbox}_{2, 1} & \\text{bbox}_{2, 2} & \\cdots & \\text{bbox}_{2, m} \\\\\n",
        "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
        "\\text{bbox}_{B, 1} & \\text{bbox}_{B, 2} & \\cdots & \\text{bbox}_{B, m} \\\\\n",
        "\\end{bmatrix}\n",
        "\\quad\n",
        "\\begin{bmatrix}\n",
        "\\text{prompt}_{1, 1} & \\text{prompt}_{1, 2} & \\cdots & \\text{prompt}_{1, m} \\\\\n",
        "\\text{prompt}_{2, 1} & \\text{prompt}_{2, 2} & \\cdots & \\text{prompt}_{2, m} \\\\\n",
        "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
        "\\text{prompt}_{B, 1} & \\text{prompt}_{B, 2} & \\cdots & \\text{prompt}_{B, m} \\\\\n",
        "\\end{bmatrix}\n",
        "$$"
      ],
      "metadata": {
        "id": "kmis3l-tReC4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Where $\\text{bbox}_{i, j}$ is the $j$th synonym of the image region delimited by bounding box $\\text{bbox}_i$ and $\\text{prompt}_j$ is the $j$th synonym of the corresponding referring expression.\n",
        "\n",
        "Unluckily, the number of bounding box and description synonyms $m$, heavily impacts the execution time and the memory consumption of the training. Our training set is populated by around 40.000 items. If we augment the data with $m=4$ synonyms for each (bounding box, text) pair, we end up with a training set with 160.000 instances. Clearly, we cannot afford the execution time required to process 160.000 examples. Furthermore, as a consequence of our data augmentation solution, the size of each batch increases. During the execution, the variables required by the calculations are stored in GPU RAM memory. The Azure GPUs have 15 Gigabytes of memory which quickly saturates with the increase of the number of synonyms $m$.\n",
        "\n",
        "At the end of the day, we did our best to deal with these limitations and we have been able to train our model for 60 epochs on our augmented dataset with $m=2$ synonyms and a batch size of 512 (that, with the $m=2$ synonyms for each ($I,P$) pair is in fact 1024). The obtained results are reported at the end of this section. Observing the learning curves, we can notice that the objective loss goes down much faster than previous solutions. Moreover, the colors on the diagonal of the matrices depicted in Figure 43, Figure 44 and Figure 45 are much brighter. Consistently, we were expected that also the evaluation on our downstream task would have been improved. On the contrary, as we can see in Table 10, the results have improved very little. To the best of our knowledge, we can formulate two reasons behind this performance evaluation outcome. First, the number of considered synonyms $m=2$ used to augment the dataset is too small to appreciate a substantial performance boost. Alternatively, it could be that it is not possible to further improve the results with this architecture. In other words, only a radical modification of the model and of the training procedure can affect the final results. Ultimately, we label this as an interesting and promising aspect to be investigated in a future expansion of this work.\n"
      ],
      "metadata": {
        "id": "tA3AJDFINRNU"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s-Y5vU4k3e_G"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/42.png)\n",
        "> \n",
        "> **Figure 43**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFjTZLxi3tVt"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/43.png)\n",
        "> \n",
        "> **Figure 44**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zuH5jMCo39Lr"
      },
      "source": [
        "> ![image.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/44.png)\n",
        "> \n",
        "> **Figure 45**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGxe8q3-7unW"
      },
      "source": [
        "> ![Screenshot 2023-09-01 at 01.24.35.png](https://raw.githubusercontent.com/gekoramy/uni.deep-learning/refs/heads/main/assets/45.png)\n",
        "> \n",
        "> **Figure 46**\n",
        "> \n",
        "> `FLYP augmented` loss"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "sI19JT-u3auZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "BATCH_SIZE: int = 1024\n",
        "LIMIT: int = -1\n",
        "EPOCHS: int = 50"
      ],
      "metadata": {
        "id": "QDj5QlC9iY5j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "split2loader: dict[Split, DataLoader[tuple[TensorImage, list[str]]]] = {\n",
        "    \"train\": DataLoader(\n",
        "        dataset=Coco4ContrastiveDataset(split=\"train\", limit=LIMIT),\n",
        "        generator=g,\n",
        "        batch_size=BATCH_SIZE,\n",
        "        collate_fn=augment,\n",
        "        shuffle=True,\n",
        "    ),\n",
        "    **{\n",
        "        split: DataLoader(\n",
        "            dataset=Coco4ContrastiveDataset(split=split, limit=LIMIT),\n",
        "            batch_size=BATCH_SIZE,\n",
        "            collate_fn=lambda x: x,\n",
        "        )\n",
        "        for split in [\"val\", \"test\"]\n",
        "    }\n",
        "}\n",
        "\n",
        "split2showtime_dataloader: dict[Split, DataLoader[tuple[TensorImage, list[str]]]] = {\n",
        "    split: DataLoader(\n",
        "        dataset=Coco4ContrastiveDataset(split=split, limit=5 * 6),\n",
        "        batch_size=6,\n",
        "        collate_fn=lambda x: x,\n",
        "        shuffle=False,\n",
        "    )\n",
        "    for split in ['train', 'val', 'test']\n",
        "}"
      ],
      "metadata": {
        "id": "CpRMXu5miZr0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def training_loop(\n",
        "        name: str,\n",
        "        model: ClipFlyp,\n",
        "        optimizer: t.Callable[[t.Iterable[torch.Tensor]], torch.optim.Optimizer],\n",
        ") -> pd.DataFrame:\n",
        "    loss: dict[str, list[float]] = defaultdict(list)\n",
        "\n",
        "    # create a logger for the experiment\n",
        "    with SummaryWriter(f\"runs/{name}\") as writer:\n",
        "        # computes evaluation results before training\n",
        "        print(\"Before training:\")\n",
        "        test_loss: float = contrastive_test_step(\n",
        "            model=model,\n",
        "            data_loader=split2loader[\"test\"],\n",
        "        )\n",
        "        val_loss: float = contrastive_test_step(\n",
        "            model=model,\n",
        "            data_loader=split2loader[\"val\"],\n",
        "        )\n",
        "\n",
        "        loss[\"test\"].append(test_loss)\n",
        "        loss[\"val\"].append(val_loss)\n",
        "\n",
        "        contrastive_showtime(\n",
        "            model,\n",
        "            split2showtime_dataloader,\n",
        "            writer,\n",
        "            0\n",
        "        )\n",
        "\n",
        "        # log to TensorBoard\n",
        "        writer.add_scalars(\n",
        "            main_tag=\"loss\",\n",
        "            tag_scalar_dict={\n",
        "                \"test\": test_loss,\n",
        "                \"val\": val_loss,\n",
        "            },\n",
        "            global_step=0,\n",
        "        )\n",
        "\n",
        "        progress = trange(EPOCHS, desc=\"EPOCHS\")\n",
        "        for epoch in progress:\n",
        "            train_loss: float = contrastive_training_step_with_synonyms(\n",
        "                model=model,\n",
        "                data_loader=split2loader[\"train\"],\n",
        "                optimizer=optimizer(model.parameters()),\n",
        "                synonyms=SYNONYMS\n",
        "            )\n",
        "\n",
        "            val_loss: float = contrastive_test_step(\n",
        "                model=model,\n",
        "                data_loader=split2loader[\"val\"],\n",
        "            )\n",
        "\n",
        "            loss[\"train\"].append(train_loss)\n",
        "            loss[\"val\"].append(val_loss)\n",
        "\n",
        "            # log to TensorBoard\n",
        "            writer.add_scalars(\n",
        "                main_tag=\"loss\",\n",
        "                tag_scalar_dict={\n",
        "                    \"train\": train_loss,\n",
        "                    \"val\": val_loss,\n",
        "                },\n",
        "                global_step=epoch + 1,\n",
        "            )\n",
        "\n",
        "            progress.set_postfix(\n",
        "                {\n",
        "                    \"train/loss\": train_loss,\n",
        "                    \"val/loss\": val_loss,\n",
        "                },\n",
        "                refresh=False,\n",
        "            )\n",
        "\n",
        "            # store model\n",
        "            torch.save(obj=model.state_dict(), f=f\"{name}-{(epoch + 1):02d}.pth\")\n",
        "\n",
        "        # compute final evaluation results\n",
        "        print(\"After training:\")\n",
        "\n",
        "        test_loss: float = contrastive_test_step(\n",
        "            model=model,\n",
        "            data_loader=split2loader[\"test\"],\n",
        "        )\n",
        "\n",
        "        loss[\"test\"].append(test_loss)\n",
        "\n",
        "        contrastive_showtime(\n",
        "            model,\n",
        "            split2showtime_dataloader,\n",
        "            writer,\n",
        "            EPOCHS\n",
        "        )\n",
        "\n",
        "        # log to TensorBoard\n",
        "        writer.add_scalars(\n",
        "            main_tag=\"loss\",\n",
        "            tag_scalar_dict={\n",
        "                \"test\": test_loss,\n",
        "            },\n",
        "            global_step=EPOCHS,\n",
        "        )\n",
        "\n",
        "        return pd.concat(\n",
        "            [\n",
        "                pd.concat(\n",
        "                    [pd.Series(v).describe() for v in loss.values()],\n",
        "                    axis=1,\n",
        "                    keys=[k for k in loss.keys()],\n",
        "                ),\n",
        "            ],\n",
        "            axis=1,\n",
        "            keys=[\"loss\"],\n",
        "        )"
      ],
      "metadata": {
        "id": "tbTw8CVnidg1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "bg_vOYJGiXpD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "keys: list[str] = [\"flyp\", \"flyp-solve-overfitting\", \"flyp-optuna\", \"flyp-augmented\"]"
      ],
      "metadata": {
        "id": "vTQS44LbgKja"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.concat(\n",
        "    [\n",
        "        pd.read_csv(f\"assets/{key}/eval-FLYP-train.csv\", index_col=0)\n",
        "        for key in keys\n",
        "    ],\n",
        "    axis=1,\n",
        "    keys=keys,\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 369
        },
        "id": "cIl1NbfGgP4L",
        "outputId": "1e021354-a1b6-4fbf-8569-9b1947a5fc98"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "               flyp                                   flyp-solve-overfitting  \\\n",
              "                iou cos similarity euclidean distance                    iou   \n",
              "count  42226.000000   42226.000000       42226.000000           42226.000000   \n",
              "mean       0.655846       0.909490           0.710470               0.610673   \n",
              "std        0.381167       0.116469           0.467298               0.391974   \n",
              "min        0.000000       0.224193           0.000000               0.000000   \n",
              "25%        0.252420       0.878059           0.371801               0.167114   \n",
              "50%        0.893520       0.966205           0.510608               0.858108   \n",
              "75%        0.952867       0.982813           0.971743               0.950501   \n",
              "max        0.998923       1.000000           3.284631               0.998923   \n",
              "\n",
              "                                          flyp-optuna                 \\\n",
              "      cos similarity euclidean distance           iou cos similarity   \n",
              "count   42226.000000       42226.000000  42226.000000   42226.000000   \n",
              "mean        0.905833           0.743905      0.616160       0.906825   \n",
              "std         0.106730           0.441283      0.389926       0.106528   \n",
              "min         0.220139           0.000000      0.000000       0.220139   \n",
              "25%         0.856192           0.387568      0.178334       0.858358   \n",
              "50%         0.958639           0.574556      0.862752       0.959355   \n",
              "75%         0.981684           1.060347      0.950845       0.981756   \n",
              "max         1.000000           2.925502      0.998923       1.000000   \n",
              "\n",
              "                         flyp-augmented                                    \n",
              "      euclidean distance            iou cos similarity euclidean distance  \n",
              "count       42226.000000   42226.000000   42226.000000       42226.000000  \n",
              "mean            0.740328       0.619415       0.906646           0.739636  \n",
              "std             0.441471       0.389647       0.107497           0.443674  \n",
              "min             0.000000       0.000000       0.220139           0.000000  \n",
              "25%             0.387047       0.183580       0.858426           0.386145  \n",
              "50%             0.568923       0.867282       0.959949           0.564835  \n",
              "75%             1.053231       0.951531       0.981854           1.053846  \n",
              "max             2.925502       0.998923       1.000000           2.790918  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bfeb2f0b-0b9e-48ad-8788-8ee49cab2e1c\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-solve-overfitting</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-optuna</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-augmented</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "      <td>42226.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.655846</td>\n",
              "      <td>0.909490</td>\n",
              "      <td>0.710470</td>\n",
              "      <td>0.610673</td>\n",
              "      <td>0.905833</td>\n",
              "      <td>0.743905</td>\n",
              "      <td>0.616160</td>\n",
              "      <td>0.906825</td>\n",
              "      <td>0.740328</td>\n",
              "      <td>0.619415</td>\n",
              "      <td>0.906646</td>\n",
              "      <td>0.739636</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.381167</td>\n",
              "      <td>0.116469</td>\n",
              "      <td>0.467298</td>\n",
              "      <td>0.391974</td>\n",
              "      <td>0.106730</td>\n",
              "      <td>0.441283</td>\n",
              "      <td>0.389926</td>\n",
              "      <td>0.106528</td>\n",
              "      <td>0.441471</td>\n",
              "      <td>0.389647</td>\n",
              "      <td>0.107497</td>\n",
              "      <td>0.443674</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.224193</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.220139</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.220139</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.220139</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.252420</td>\n",
              "      <td>0.878059</td>\n",
              "      <td>0.371801</td>\n",
              "      <td>0.167114</td>\n",
              "      <td>0.856192</td>\n",
              "      <td>0.387568</td>\n",
              "      <td>0.178334</td>\n",
              "      <td>0.858358</td>\n",
              "      <td>0.387047</td>\n",
              "      <td>0.183580</td>\n",
              "      <td>0.858426</td>\n",
              "      <td>0.386145</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.893520</td>\n",
              "      <td>0.966205</td>\n",
              "      <td>0.510608</td>\n",
              "      <td>0.858108</td>\n",
              "      <td>0.958639</td>\n",
              "      <td>0.574556</td>\n",
              "      <td>0.862752</td>\n",
              "      <td>0.959355</td>\n",
              "      <td>0.568923</td>\n",
              "      <td>0.867282</td>\n",
              "      <td>0.959949</td>\n",
              "      <td>0.564835</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.952867</td>\n",
              "      <td>0.982813</td>\n",
              "      <td>0.971743</td>\n",
              "      <td>0.950501</td>\n",
              "      <td>0.981684</td>\n",
              "      <td>1.060347</td>\n",
              "      <td>0.950845</td>\n",
              "      <td>0.981756</td>\n",
              "      <td>1.053231</td>\n",
              "      <td>0.951531</td>\n",
              "      <td>0.981854</td>\n",
              "      <td>1.053846</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.998923</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.284631</td>\n",
              "      <td>0.998923</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.925502</td>\n",
              "      <td>0.998923</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.925502</td>\n",
              "      <td>0.998923</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.790918</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bfeb2f0b-0b9e-48ad-8788-8ee49cab2e1c')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-bfeb2f0b-0b9e-48ad-8788-8ee49cab2e1c button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-bfeb2f0b-0b9e-48ad-8788-8ee49cab2e1c');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3aa4939a-feb8-432e-946b-8a369a50f3c6\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3aa4939a-feb8-432e-946b-8a369a50f3c6')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3aa4939a-feb8-432e-946b-8a369a50f3c6 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 91
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 6**"
      ],
      "metadata": {
        "id": "NoziVSAkPhEh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.concat(\n",
        "    [\n",
        "        pd.read_csv(f\"assets/{key}/eval-FLYP-val.csv\", index_col=0)\n",
        "        for key in keys\n",
        "    ],\n",
        "    axis=1,\n",
        "    keys=keys,\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "id": "T_5gJCP1gRQt",
        "outputId": "aba4bcff-ad49-48ca-c83a-80f62caa4269"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              flyp                                   flyp-solve-overfitting  \\\n",
              "               iou cos similarity euclidean distance                    iou   \n",
              "count  2573.000000    2573.000000        2573.000000            2573.000000   \n",
              "mean      0.444740       0.845191           0.964074               0.559027   \n",
              "std       0.411338       0.147209           0.537300               0.400013   \n",
              "min       0.000000       0.310306           0.070107               0.000000   \n",
              "25%       0.009099       0.757501           0.457350               0.102296   \n",
              "50%       0.305549       0.890930           0.922397               0.724764   \n",
              "75%       0.921113       0.972397           1.349703               0.944698   \n",
              "max       0.993985       0.999632           2.726751               0.994411   \n",
              "\n",
              "                                         flyp-optuna                 \\\n",
              "      cos similarity euclidean distance          iou cos similarity   \n",
              "count    2573.000000        2573.000000  2573.000000    2573.000000   \n",
              "mean        0.892097           0.802483     0.566936       0.892032   \n",
              "std         0.113741           0.457968     0.398357       0.115519   \n",
              "min         0.365210           0.057728     0.000000       0.365210   \n",
              "25%         0.826868           0.411174     0.109141       0.831729   \n",
              "50%         0.941753           0.677477     0.754302       0.943839   \n",
              "75%         0.979479           1.143081     0.945063       0.979501   \n",
              "max         0.999632           2.474040     0.994411       0.999632   \n",
              "\n",
              "                         flyp-augmented                                    \n",
              "      euclidean distance            iou cos similarity euclidean distance  \n",
              "count        2573.000000    2573.000000    2573.000000        2573.000000  \n",
              "mean            0.802094       0.561768       0.890657           0.807573  \n",
              "std             0.462112       0.397323       0.116169           0.464711  \n",
              "min             0.057728       0.000000       0.365210           0.057728  \n",
              "25%             0.411081       0.112111       0.827230           0.410835  \n",
              "50%             0.663102       0.724764       0.941327           0.677092  \n",
              "75%             1.137241       0.944240       0.979479           1.150478  \n",
              "max             2.474040       0.993985       0.999632           2.474040  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9b0e0d70-1268-4928-8811-6d96e4c6f7ad\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-solve-overfitting</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-optuna</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-augmented</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "      <td>2573.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.444740</td>\n",
              "      <td>0.845191</td>\n",
              "      <td>0.964074</td>\n",
              "      <td>0.559027</td>\n",
              "      <td>0.892097</td>\n",
              "      <td>0.802483</td>\n",
              "      <td>0.566936</td>\n",
              "      <td>0.892032</td>\n",
              "      <td>0.802094</td>\n",
              "      <td>0.561768</td>\n",
              "      <td>0.890657</td>\n",
              "      <td>0.807573</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.411338</td>\n",
              "      <td>0.147209</td>\n",
              "      <td>0.537300</td>\n",
              "      <td>0.400013</td>\n",
              "      <td>0.113741</td>\n",
              "      <td>0.457968</td>\n",
              "      <td>0.398357</td>\n",
              "      <td>0.115519</td>\n",
              "      <td>0.462112</td>\n",
              "      <td>0.397323</td>\n",
              "      <td>0.116169</td>\n",
              "      <td>0.464711</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.310306</td>\n",
              "      <td>0.070107</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.365210</td>\n",
              "      <td>0.057728</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.365210</td>\n",
              "      <td>0.057728</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.365210</td>\n",
              "      <td>0.057728</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.009099</td>\n",
              "      <td>0.757501</td>\n",
              "      <td>0.457350</td>\n",
              "      <td>0.102296</td>\n",
              "      <td>0.826868</td>\n",
              "      <td>0.411174</td>\n",
              "      <td>0.109141</td>\n",
              "      <td>0.831729</td>\n",
              "      <td>0.411081</td>\n",
              "      <td>0.112111</td>\n",
              "      <td>0.827230</td>\n",
              "      <td>0.410835</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.305549</td>\n",
              "      <td>0.890930</td>\n",
              "      <td>0.922397</td>\n",
              "      <td>0.724764</td>\n",
              "      <td>0.941753</td>\n",
              "      <td>0.677477</td>\n",
              "      <td>0.754302</td>\n",
              "      <td>0.943839</td>\n",
              "      <td>0.663102</td>\n",
              "      <td>0.724764</td>\n",
              "      <td>0.941327</td>\n",
              "      <td>0.677092</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.921113</td>\n",
              "      <td>0.972397</td>\n",
              "      <td>1.349703</td>\n",
              "      <td>0.944698</td>\n",
              "      <td>0.979479</td>\n",
              "      <td>1.143081</td>\n",
              "      <td>0.945063</td>\n",
              "      <td>0.979501</td>\n",
              "      <td>1.137241</td>\n",
              "      <td>0.944240</td>\n",
              "      <td>0.979479</td>\n",
              "      <td>1.150478</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.993985</td>\n",
              "      <td>0.999632</td>\n",
              "      <td>2.726751</td>\n",
              "      <td>0.994411</td>\n",
              "      <td>0.999632</td>\n",
              "      <td>2.474040</td>\n",
              "      <td>0.994411</td>\n",
              "      <td>0.999632</td>\n",
              "      <td>2.474040</td>\n",
              "      <td>0.993985</td>\n",
              "      <td>0.999632</td>\n",
              "      <td>2.474040</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9b0e0d70-1268-4928-8811-6d96e4c6f7ad')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9b0e0d70-1268-4928-8811-6d96e4c6f7ad button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9b0e0d70-1268-4928-8811-6d96e4c6f7ad');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7ad9b108-9561-4b3b-ba44-9c6a2c39d6c2\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7ad9b108-9561-4b3b-ba44-9c6a2c39d6c2')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7ad9b108-9561-4b3b-ba44-9c6a2c39d6c2 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 7**"
      ],
      "metadata": {
        "id": "J62TT8CmPoPK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.concat(\n",
        "    [\n",
        "        pd.read_csv(f\"assets/{key}/eval-FLYP-test.csv\", index_col=0)\n",
        "        for key in keys\n",
        "    ],\n",
        "    axis=1,\n",
        "    keys=keys,\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "id": "gYoXxdWogSpO",
        "outputId": "28315f8b-c580-418b-bea0-4745642a516c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              flyp                                   flyp-solve-overfitting  \\\n",
              "               iou cos similarity euclidean distance                    iou   \n",
              "count  5023.000000    5023.000000        5023.000000            5023.000000   \n",
              "mean      0.451015       0.846404           0.961283               0.580078   \n",
              "std       0.410500       0.147500           0.545707               0.397923   \n",
              "min       0.000000       0.263428           0.000000               0.000000   \n",
              "25%       0.013494       0.753941           0.450327               0.122994   \n",
              "50%       0.326402       0.891457           0.921334               0.797624   \n",
              "75%       0.924748       0.974087           1.360280               0.948823   \n",
              "max       0.998143       1.000000           2.888018               0.996050   \n",
              "\n",
              "                                         flyp-optuna                 \\\n",
              "      cos similarity euclidean distance          iou cos similarity   \n",
              "count    5023.000000        5023.000000  5023.000000    5023.000000   \n",
              "mean        0.896025           0.785909     0.577689       0.895028   \n",
              "std         0.113407           0.462311     0.398581       0.115062   \n",
              "min         0.342195           0.000000     0.000000       0.342195   \n",
              "25%         0.836112           0.397114     0.122399       0.834798   \n",
              "50%         0.951409           0.630213     0.794154       0.950434   \n",
              "75%         0.980802           1.124640     0.948546       0.980756   \n",
              "max         1.000000           2.777238     0.995533       1.000000   \n",
              "\n",
              "                         flyp-augmented                                    \n",
              "      euclidean distance            iou cos similarity euclidean distance  \n",
              "count        5023.000000    5023.000000    5023.000000        5023.000000  \n",
              "mean            0.789919       0.576252       0.893777           0.794152  \n",
              "std             0.466610       0.398819       0.116289           0.469464  \n",
              "min             0.000000       0.000000       0.280131           0.000000  \n",
              "25%             0.397960       0.119033       0.832247           0.399706  \n",
              "50%             0.631388       0.788843       0.950299           0.631416  \n",
              "75%             1.128206       0.948779       0.980454           1.138750  \n",
              "max             2.794651       0.996050       1.000000           2.565383  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9190cef4-d65c-4ffc-98c5-471e5e16a5fd\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-solve-overfitting</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-optuna</th>\n",
              "      <th colspan=\"3\" halign=\"left\">flyp-augmented</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "      <th>iou</th>\n",
              "      <th>cos similarity</th>\n",
              "      <th>euclidean distance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "      <td>5023.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.451015</td>\n",
              "      <td>0.846404</td>\n",
              "      <td>0.961283</td>\n",
              "      <td>0.580078</td>\n",
              "      <td>0.896025</td>\n",
              "      <td>0.785909</td>\n",
              "      <td>0.577689</td>\n",
              "      <td>0.895028</td>\n",
              "      <td>0.789919</td>\n",
              "      <td>0.576252</td>\n",
              "      <td>0.893777</td>\n",
              "      <td>0.794152</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.410500</td>\n",
              "      <td>0.147500</td>\n",
              "      <td>0.545707</td>\n",
              "      <td>0.397923</td>\n",
              "      <td>0.113407</td>\n",
              "      <td>0.462311</td>\n",
              "      <td>0.398581</td>\n",
              "      <td>0.115062</td>\n",
              "      <td>0.466610</td>\n",
              "      <td>0.398819</td>\n",
              "      <td>0.116289</td>\n",
              "      <td>0.469464</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.263428</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.342195</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.342195</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.280131</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.013494</td>\n",
              "      <td>0.753941</td>\n",
              "      <td>0.450327</td>\n",
              "      <td>0.122994</td>\n",
              "      <td>0.836112</td>\n",
              "      <td>0.397114</td>\n",
              "      <td>0.122399</td>\n",
              "      <td>0.834798</td>\n",
              "      <td>0.397960</td>\n",
              "      <td>0.119033</td>\n",
              "      <td>0.832247</td>\n",
              "      <td>0.399706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.326402</td>\n",
              "      <td>0.891457</td>\n",
              "      <td>0.921334</td>\n",
              "      <td>0.797624</td>\n",
              "      <td>0.951409</td>\n",
              "      <td>0.630213</td>\n",
              "      <td>0.794154</td>\n",
              "      <td>0.950434</td>\n",
              "      <td>0.631388</td>\n",
              "      <td>0.788843</td>\n",
              "      <td>0.950299</td>\n",
              "      <td>0.631416</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.924748</td>\n",
              "      <td>0.974087</td>\n",
              "      <td>1.360280</td>\n",
              "      <td>0.948823</td>\n",
              "      <td>0.980802</td>\n",
              "      <td>1.124640</td>\n",
              "      <td>0.948546</td>\n",
              "      <td>0.980756</td>\n",
              "      <td>1.128206</td>\n",
              "      <td>0.948779</td>\n",
              "      <td>0.980454</td>\n",
              "      <td>1.138750</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.998143</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.888018</td>\n",
              "      <td>0.996050</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.777238</td>\n",
              "      <td>0.995533</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.794651</td>\n",
              "      <td>0.996050</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.565383</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9190cef4-d65c-4ffc-98c5-471e5e16a5fd')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9190cef4-d65c-4ffc-98c5-471e5e16a5fd button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9190cef4-d65c-4ffc-98c5-471e5e16a5fd');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7ac56a10-ca70-474a-8bec-e3dec64bcd95\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7ac56a10-ca70-474a-8bec-e3dec64bcd95')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7ac56a10-ca70-474a-8bec-e3dec64bcd95 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 8**"
      ],
      "metadata": {
        "id": "A6yAUhHgPqK3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.read_csv(\"assets/comparing[train].csv\", index_col=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "gGfhBvUGgUFc",
        "outputId": "b48874b1-4d10-4c46-df3f-8ac113dfe7c8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        mAP[IoU .3]  mAP[IoU .5]  mAP[IoU .7]      mIoU  \\\n",
              "FLYP                       0.735542     0.686875     0.638232  0.655846   \n",
              "FLYP solve overfitting     0.691967     0.628712     0.573509  0.610673   \n",
              "FLYP optuna                0.698385     0.635248     0.579809  0.616160   \n",
              "FLYP augmented             0.700374     0.639345     0.584142  0.619415   \n",
              "\n",
              "                            mCos       mED  \n",
              "FLYP                    0.909490  0.710470  \n",
              "FLYP solve overfitting  0.905833  0.743905  \n",
              "FLYP optuna             0.906825  0.740328  \n",
              "FLYP augmented          0.906646  0.739636  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c3ebf6aa-c921-4f4e-8512-ada5da3c809e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mAP[IoU .3]</th>\n",
              "      <th>mAP[IoU .5]</th>\n",
              "      <th>mAP[IoU .7]</th>\n",
              "      <th>mIoU</th>\n",
              "      <th>mCos</th>\n",
              "      <th>mED</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>FLYP</th>\n",
              "      <td>0.735542</td>\n",
              "      <td>0.686875</td>\n",
              "      <td>0.638232</td>\n",
              "      <td>0.655846</td>\n",
              "      <td>0.909490</td>\n",
              "      <td>0.710470</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP solve overfitting</th>\n",
              "      <td>0.691967</td>\n",
              "      <td>0.628712</td>\n",
              "      <td>0.573509</td>\n",
              "      <td>0.610673</td>\n",
              "      <td>0.905833</td>\n",
              "      <td>0.743905</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP optuna</th>\n",
              "      <td>0.698385</td>\n",
              "      <td>0.635248</td>\n",
              "      <td>0.579809</td>\n",
              "      <td>0.616160</td>\n",
              "      <td>0.906825</td>\n",
              "      <td>0.740328</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP augmented</th>\n",
              "      <td>0.700374</td>\n",
              "      <td>0.639345</td>\n",
              "      <td>0.584142</td>\n",
              "      <td>0.619415</td>\n",
              "      <td>0.906646</td>\n",
              "      <td>0.739636</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c3ebf6aa-c921-4f4e-8512-ada5da3c809e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c3ebf6aa-c921-4f4e-8512-ada5da3c809e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c3ebf6aa-c921-4f4e-8512-ada5da3c809e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e98a19b1-91d8-49f7-93df-9792ebfcedeb\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e98a19b1-91d8-49f7-93df-9792ebfcedeb')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e98a19b1-91d8-49f7-93df-9792ebfcedeb button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 9**"
      ],
      "metadata": {
        "id": "z1B3SnfAPrhs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.read_csv(\"assets/comparing[val].csv\", index_col=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "2gAD4HKEgVMw",
        "outputId": "14ecc8f5-c607-436d-d879-57d38c650a42"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        mAP[IoU .3]  mAP[IoU .5]  mAP[IoU .7]      mIoU  \\\n",
              "FLYP                       0.501749     0.447338     0.395647  0.444740   \n",
              "FLYP solve overfitting     0.640497     0.570152     0.511465  0.559027   \n",
              "FLYP optuna                0.649048     0.582588     0.519627  0.566936   \n",
              "FLYP augmented             0.643995     0.573649     0.511465  0.561768   \n",
              "\n",
              "                            mCos       mED  \n",
              "FLYP                    0.845191  0.964074  \n",
              "FLYP solve overfitting  0.892097  0.802483  \n",
              "FLYP optuna             0.892032  0.802094  \n",
              "FLYP augmented          0.890657  0.807573  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-953787ce-418a-407b-a746-5e806d366c45\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mAP[IoU .3]</th>\n",
              "      <th>mAP[IoU .5]</th>\n",
              "      <th>mAP[IoU .7]</th>\n",
              "      <th>mIoU</th>\n",
              "      <th>mCos</th>\n",
              "      <th>mED</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>FLYP</th>\n",
              "      <td>0.501749</td>\n",
              "      <td>0.447338</td>\n",
              "      <td>0.395647</td>\n",
              "      <td>0.444740</td>\n",
              "      <td>0.845191</td>\n",
              "      <td>0.964074</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP solve overfitting</th>\n",
              "      <td>0.640497</td>\n",
              "      <td>0.570152</td>\n",
              "      <td>0.511465</td>\n",
              "      <td>0.559027</td>\n",
              "      <td>0.892097</td>\n",
              "      <td>0.802483</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP optuna</th>\n",
              "      <td>0.649048</td>\n",
              "      <td>0.582588</td>\n",
              "      <td>0.519627</td>\n",
              "      <td>0.566936</td>\n",
              "      <td>0.892032</td>\n",
              "      <td>0.802094</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP augmented</th>\n",
              "      <td>0.643995</td>\n",
              "      <td>0.573649</td>\n",
              "      <td>0.511465</td>\n",
              "      <td>0.561768</td>\n",
              "      <td>0.890657</td>\n",
              "      <td>0.807573</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-953787ce-418a-407b-a746-5e806d366c45')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-953787ce-418a-407b-a746-5e806d366c45 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-953787ce-418a-407b-a746-5e806d366c45');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6a2c0192-fe6c-48ae-ba82-e373f320c6ac\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6a2c0192-fe6c-48ae-ba82-e373f320c6ac')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6a2c0192-fe6c-48ae-ba82-e373f320c6ac button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 10**"
      ],
      "metadata": {
        "id": "80sw1f0WPspu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.read_csv(\"assets/comparing[test].csv\", index_col=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "bPuJ5wwggXqb",
        "outputId": "1ae22b0b-6d2e-4d63-e973-881ebef97e9e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        mAP[IoU .3]  mAP[IoU .5]  mAP[IoU .7]      mIoU  \\\n",
              "FLYP                       0.511447     0.448139     0.394983  0.451015   \n",
              "FLYP solve overfitting     0.660960     0.591678     0.532749  0.580078   \n",
              "FLYP optuna                0.655385     0.588891     0.530958  0.577689   \n",
              "FLYP augmented             0.654788     0.588095     0.528967  0.576252   \n",
              "\n",
              "                            mCos       mED  \n",
              "FLYP                    0.846404  0.961283  \n",
              "FLYP solve overfitting  0.896025  0.785909  \n",
              "FLYP optuna             0.895028  0.789919  \n",
              "FLYP augmented          0.893777  0.794152  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3f5ad2e4-5535-428a-8a56-ba68b3659ebf\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mAP[IoU .3]</th>\n",
              "      <th>mAP[IoU .5]</th>\n",
              "      <th>mAP[IoU .7]</th>\n",
              "      <th>mIoU</th>\n",
              "      <th>mCos</th>\n",
              "      <th>mED</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>FLYP</th>\n",
              "      <td>0.511447</td>\n",
              "      <td>0.448139</td>\n",
              "      <td>0.394983</td>\n",
              "      <td>0.451015</td>\n",
              "      <td>0.846404</td>\n",
              "      <td>0.961283</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP solve overfitting</th>\n",
              "      <td>0.660960</td>\n",
              "      <td>0.591678</td>\n",
              "      <td>0.532749</td>\n",
              "      <td>0.580078</td>\n",
              "      <td>0.896025</td>\n",
              "      <td>0.785909</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP optuna</th>\n",
              "      <td>0.655385</td>\n",
              "      <td>0.588891</td>\n",
              "      <td>0.530958</td>\n",
              "      <td>0.577689</td>\n",
              "      <td>0.895028</td>\n",
              "      <td>0.789919</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>FLYP augmented</th>\n",
              "      <td>0.654788</td>\n",
              "      <td>0.588095</td>\n",
              "      <td>0.528967</td>\n",
              "      <td>0.576252</td>\n",
              "      <td>0.893777</td>\n",
              "      <td>0.794152</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3f5ad2e4-5535-428a-8a56-ba68b3659ebf')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3f5ad2e4-5535-428a-8a56-ba68b3659ebf button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3f5ad2e4-5535-428a-8a56-ba68b3659ebf');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b937f58c-6bc2-4702-8112-c121250212ce\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b937f58c-6bc2-4702-8112-c121250212ce')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b937f58c-6bc2-4702-8112-c121250212ce button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Table 11**"
      ],
      "metadata": {
        "id": "TtnzmqAgPuE4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "13.5 Weight initialization"
      ],
      "metadata": {
        "id": "p1anhl4wJikB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In addition to the aforementioned strategies, a tentative which is worth mentioning is the change of the weight initialization algorithm. More in depth, we have implemented the popular Xavier initialization as originally proposed by Xavier Glorot and Yoshua Bengio in their work \"Understanding the difficulty of training deep feedforward neural networks\" [34]. However, in line with [this post](https://stats.stackexchange.com/a/319849) on stackexchange.com, we have experimentally noticed that the PyTorch default Kaiming initialization [35] performs considerably better in our case. Hence, we have not further investigated this aspect."
      ],
      "metadata": {
        "id": "yFzwYzZmNbDB"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QK_DYB4qn-46"
      },
      "source": [
        "## 14 Conclusion and further research directions"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this work we have built and evaluated several deep learning frameworks on top of the zero-shot capabilities of CLIP to perform visual grounding on the RefCOCOg dataset. To this end, motivated by other related works (Section 4), we have presented a two steps joint embedding approach to tackle the problem.\n",
        "\n",
        "Given an input image, the purpose of the first step is to propose a set of regions delimited by rectangular bounding boxes containing potentially relevant objects. In order to accomplish this we have compared several modern object detection algorithms. In this regard, as an important contribution, we have made available three new versions of the RefCOCOg dataset filled with the bounding boxes predicted by Yolov5, Yolov8 and DETR respectively, together with the code readily available to be customized for other arbitrary object detection algorithms. With this preprocessing we increase the number of iterations per second of our training loop by the 70 percent. We believe that this resource can be helpful for many other future research directions.\n",
        "\n",
        "The proposed solution to address the second step of the of the overall procedure can be divided in four categories:\n",
        "* training-free approach\n",
        "* standard fine-tuning approach\n",
        "* contrastive learning approach\n",
        "* self-attention approach\n",
        "\n",
        "All of them have been evaluated with commonly used metrics on their ability to ground textual descriptions on the visual world. As an important contribution of this notebook we show that despite its simplicity, fine-tuning the CLIP image and textual encoders following the same pretraining contrastive learning pattern proposed by the original authors, consistently outperforms alternative approaches.\n",
        "\n",
        "Unfortunately, we have not been able due to our limited computational resources to evaluate our self-attention model on a proper amount of data for a proper amount of time. We leave this as a prominent and interesting future research direction.\n",
        "\n",
        "Finally, in Section 13, we have further refined the generalization capabilities of our most promising architecture by means of Optuna automatic hyperparameter optimization tool, data augmentation, and noise injection.\n",
        "\n",
        "Last but not least, we have remarkably provided in this document the Python code implementation of every commented aspect."
      ],
      "metadata": {
        "id": "vbrfvezROHey"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We conclude our project with some final valuable future research direction which might inspire further research on this topic.\n",
        "\n",
        "In our opinion the first stage of our two steps joint embedding approach is the component with the largest room of improvement. In order to find the regions of the image which reflect the most the content of a textual description we, as humans, do not extract all the relevant objects which populates the field of view as a prior step. Rather, the attention of our cognitive system is guided by the referring expression to immediately preclude portions of the image which are not compatible with the input query. In this regard, we suggest to explore the literature about text-guided attention models as a good starting point to improve our framework [36][37].\n",
        "\n",
        "The most challenging problem which remains substantially unsolved by this work concerns the presence of spatial relationships, like \"The fruit in the middle\", in the natural language descriptions. Our self-attention approach detailed in Section 12 partially overcomes this limitation. However, given the frequency with which location words are used in real world applications, this topic definitely deserves further investigation in the future."
      ],
      "metadata": {
        "id": "ZvEq57CCO5nq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "> ***Remark:***\n",
        "> *in order to systematically keep track of the experiments that we have made throughout the development of the project and to convenitenly visualize their corresponsing outcomes we have used TensorBoard. Unfortunately, we can not deliver the notebook with the beautiful interactive TensorBoard dashboard. Indeed, the resulting file would be to big to be sent via email.*"
      ],
      "metadata": {
        "id": "tjaKJb1TJJ5a"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9WFhWwKHsmNo"
      },
      "source": [
        "## 15 References"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOBHVFxT-XSI"
      },
      "source": [
        "[[1]](https://www.cs.utexas.edu/users/ai-lab/downloadPublication.php?filename=http://www.cs.utexas.edu/users/ml/papers/thomason.robonlp17.pdf&pubid=127642)  Jesse Thomason, Jivko Sinapov, and Raymond Mooney,\n",
        "\"Guiding interaction behaviors for multi-modal grounded language learning,\" in Proceedings of the First Workshop on Language Grounding for Robotics, 2017"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10BteNVyIY_Y"
      },
      "source": [
        "[[2]](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "otRUfAy3IvGQ"
      },
      "source": [
        "[[3]](https://arxiv.org/abs/1608.00272) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zteDkUBb-kPz"
      },
      "source": [
        "[[4]](https://arxiv.org/abs/2007.09554) Yanyuan Qiao and Chaorui Deng and Qi Wu. Referring Expression Comprehension: A Survey of Methods and Datasets. Year 2020."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ccJESVMo1LtB"
      },
      "source": [
        "[[5]](https://arxiv.org/abs/2212.00638) Sachin Goyal and Ananya Kumar and Sankalp Garg and Zico Kolter and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. Year 2022."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gVXjpgAG9qFl"
      },
      "source": [
        "[[6]](https://ieeexplore.ieee.org/document/8845685) S. Qiu, Y. Zhao, J. Jiao, Y. Wei, and S. Wei, \"Referring image segmentation by generative adversarial learning\", IEEE Trans. Multimedia. Year 2020."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QxFq8Xod-CGo"
      },
      "source": [
        "[[7]](https://arxiv.org/abs/1505.00468) S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick,\n",
        "and D. Parikh, \"Vqa: Visual question answering\", in Proc. IEEE Int.\n",
        "Conf. Comput. Vis. Year 2015."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-2WVLWYC-Nbo"
      },
      "source": [
        "[[8]](https://ieeexplore.ieee.org/document/9422035) Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel, \"Visual question answering: A survey of methods and datasets\" Comput.\n",
        "Vis. Image Underst. Year 2017."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nc1dT728-qh4"
      },
      "source": [
        "[[9]](https://arxiv.org/abs/1904.05548) Z. Zheng, W. Wang, S. Qi, and S. Zhu, \"Reasoning visual dialogs with structural and partial observations\" in Proc. IEEE Conf. Comput. Vis.\n",
        "Pattern Recognit. Year 2019."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FODQUqbE-2D2"
      },
      "source": [
        "[[10]](https://arxiv.org/abs/1809.01816) S. Kottur, J. M. F. Moura, D. Parikh, D. Batra, and M. Rohrbach, \"Visual coreference resolution in visual dialog using neural module networks\",\n",
        "in Proc. Eur. Conf. Comput. Vis. Year 2018."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7jZmkDdw_cLI"
      },
      "source": [
        "[[11]](https://arxiv.org/abs/1511.02283) J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, \"Generation and comprehension of unambiguous object descriptions\", in\n",
        "Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Year 2016."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P9ig1Ijn_3a1"
      },
      "source": [
        "[[12]](https://arxiv.org/abs/2202.10054) Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. \"Fine-tuning can distort\n",
        "pretrained features and underperform out-of-distribution\". In International Conference on Learning Representations\n",
        "(ICLR). Year 2022."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yu0fnbLB__F4"
      },
      "source": [
        "[[13]](https://arxiv.org/abs/2109.01903) Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok\n",
        "Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models.CoRR. Year 2021."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CQJGUkyyzNev"
      },
      "source": [
        "[[14]](https://arxiv.org/abs/1803.01534) Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. Year 2018."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2TQ_I-cZsZQX"
      },
      "source": [
        "[[15]](https://arxiv.org/abs/2104.11892) Syed Sahil Abbas Zaidi and Mohammad Samar Ansari and Asra Aslam and Nadia Kanwal and Mamoona Asghar and Brian Lee. A Survey of Modern Deep Learning based Object Detection Models. Year 2021."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OkRSG7zG_zZW"
      },
      "source": [
        "[[16]](https://paperswithcode.com/task/object-detection) paperswithcode.com list of state of the art Object Detection algorithms. Last visited on 28th August 2023."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FsNTHZixAHFP"
      },
      "source": [
        "[[17]](https://arxiv.org/abs/1506.02640) Joseph Redmon and Santosh Divvala and Ross Girshick and Ali Farhadi. You Only Look Once: Unified, Real-Time Object Detection. Year 2015."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tTBbL75MAPMm"
      },
      "source": [
        "[[18]](https://arxiv.org/abs/2005.12872) Nicolas Carion and Francisco Massa and Gabriel Synnaeve and Nicolas Usunier and Alexander Kirillov and Sergey Zagoruyko. End-to-End Object Detection with Transformers. Year 2020."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8r5uWIDY7Zi-"
      },
      "source": [
        "[[19]](https://arxiv.org/abs/2109.01903) Mitchell Wortsman and Gabriel Ilharco and Jong Wook Kim and Mike Li and Simon Kornblith and Rebecca Roelofs and Raphael Gontijo-Lopes and Hannaneh Hajishirzi and Ali Farhadi and Hongseok Namkoong and Ludwig Schmidt. Robust fine-tuning of zero-shot models. Year 2021."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QwjZ9p4j7oE9"
      },
      "source": [
        "[[20]](https://arxiv.org/abs/2308.12919) Jian Liang and Lijun Sheng and Zhengbo Wang and Ran He and Tieniu Tan. Towards Realistic Unsupervised Fine-tuning with CLIP. Year 2023."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G6FKc0o870Qu"
      },
      "source": [
        "[[21]](https://arxiv.org/abs/2212.03640) Hanoona Rasheed and Muhammad Uzair Khattak and Muhammad Maaz and Salman Khan and Fahad Shahbaz Khan. Fine-tuned CLIP Models are Efficient Video Learners Year 2022."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4xC_WlYV8C5G"
      },
      "source": [
        "[[22]](https://arxiv.org/abs/2007.03051) Lasse F. Wolff Anthony and Benjamin Kanding and Raghavendra Selvan. Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models. Year 2020."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eb3IoDW7qCBg"
      },
      "source": [
        "[[23]](https://arxiv.org/abs/2101.06983) Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan. Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup. Year 2021."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gve1rNt2_oWE"
      },
      "source": [
        "[[24]](https://arxiv.org/abs/1706.03762) Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin. Attention Is All You Need. Year 2017."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[25]](https://arxiv.org/abs/2304.11127) Shuhei Watanabe. Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance. Year 2023."
      ],
      "metadata": {
        "id": "7QkBq17UIoBZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[26]](https://arxiv.org/abs/1603.06560) Lisha Li and Kevin Jamieson and Giulia DeSalvo and Afshin Rostamizadeh and Ameet Talwalkar. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Year 2016."
      ],
      "metadata": {
        "id": "WBf9IqUUI1PR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[27]](https://arxiv.org/abs/2005.14165) Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei. Language Models are Few-Shot Learners. Year 2020."
      ],
      "metadata": {
        "id": "IsT20dh5KZcb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[28]](https://arxiv.org/abs/2012.15723) Tianyu Gao and Adam Fisch and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. Year 2020."
      ],
      "metadata": {
        "id": "UPR9Wvz_KqpS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[29]](https://arxiv.org/abs/1907.03752) Marivate, Vukosi and Sefara, Tshephisho. Improving Short Text Classification Through Global Augmentation Methods. Year 2020.\n"
      ],
      "metadata": {
        "id": "p3QpoycuK82R"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[30]](https://aclanthology.org/D19-1670.pdf) Jason Wei, Kai Zou. EDA: Easy Data Augmentation Techniques for Boosting Performance on\n",
        "Text Classification Tasks. Year 2019."
      ],
      "metadata": {
        "id": "FXfzX_UuLKJT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[31]](https://arxiv.org/abs/1912.08777) Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. Year 2019."
      ],
      "metadata": {
        "id": "zFVNJLz5LiZm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[32]](https://arxiv.org/abs/1910.13461) Mike Lewis and Yinhan Liu and Naman Goyal and Marjan Ghazvininejad and Abdelrahman Mohamed and Omer Levy and Ves Stoyanov and Luke Zettlemoyer. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Year 2019."
      ],
      "metadata": {
        "id": "uppRrHNKLwn5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[33]](https://doi.org/10.1162/neco.1997.9.5.1093) Yves Grandvalet, St\u00e9phane Canu, St\u00e9phane Boucheron. Noise Injection: Theoretical Prospects. Year 1997."
      ],
      "metadata": {
        "id": "p3iV4FTSMf-C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[34]](https://paperswithcode.com/paper/understanding-the-difficulty-of-training-deep) Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Year 2010."
      ],
      "metadata": {
        "id": "3AxfxV8GNfl4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[35]](https://arxiv.org/abs/1502.01852v1) Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Year 2015."
      ],
      "metadata": {
        "id": "aSYoHvztNvMO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[36]](https://ieeexplore.ieee.org/document/9207417) L. Zhang, X. Wang, L. Yao and F. Zheng. \"Zero-Shot Object Detection with Textual Descriptions Using Convolutional Neural Networks. Year 2020."
      ],
      "metadata": {
        "id": "E084Fc8cOcYJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[[37]](https://arxiv.org/abs/1612.03557) Jonghwan Mun and Minsu Cho and Bohyung Han. Text-guided Attention Model for Image Captioning. Year 2016."
      ],
      "metadata": {
        "id": "s5avGmqiOmqq"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}