Siamese_Models.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Siamese Models.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyPLJK0egil14XDJi/ioHbc7"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "y5jrfG5rND_v"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "from keras.layers.embeddings import Embedding\n",
        "\n",
        "from keras.utils import np_utils\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, GlobalAveragePooling1D, Flatten, Lambda\n",
        "from keras import backend as K\n",
        "\n",
        "from keras import Input\n",
        "from keras.models import Model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lHja0PWINarM"
      },
      "source": [
        "'''\n",
        "Since we are dealing with tensors, we cannot directly use sum/pow etc functions.\n",
        "Fot this tensorflow has a library called backend where we can find all sorts of\n",
        "function to operate on tensors\n",
        "'''\n",
        "\n",
        "def euclidean_distance(vects):\n",
        "    x, y = vects\n",
        "    sum_square = K.sum(K.square(x - y), axis=1, keepdims=True)\n",
        "    return K.sqrt(K.maximum(sum_square, K.epsilon()))\n",
        "\n",
        "def eucl_dist_output_shape(shapes):\n",
        "    shape1, shape2 = shapes\n",
        "    return (shape1[0], 1)\n",
        "\n",
        "\n",
        "def contrastive_loss(y_true, y_pred):\n",
        "    '''Contrastive loss from Hadsell-et-al.'06\n",
        "    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf\n",
        "    '''\n",
        "    margin = 1\n",
        "    square_pred = K.square(y_pred)\n",
        "    margin_square = K.square(K.maximum(margin - y_pred, 0))\n",
        "\n",
        "    return K.mean(y_true * square_pred + (1 - y_true) * margin_square)\n",
        "\n",
        "def accuracy(y_true, y_pred):\n",
        "    '''Compute classification accuracy with a fixed threshold on distances.\n",
        "    '''\n",
        "    return K.mean(K.equal(y_true, K.cast(y_pred < 0.5, y_true.dtype)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8b9qsN4bN_Vk"
      },
      "source": [
        "### **Training 1 network for Different Inputs**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCbq1DVrVaKB"
      },
      "source": [
        "\n",
        "\n",
        "1.   Siamese means twin, this architecture takes in 2 inputs, passes it from one model and outputs the embeddings\n",
        "2.   The 2 inputs can be an image or text inputs or any vector\n",
        "3. After we get the embeddings, we calculate the distances between them and using a loss function either contrastive or triplet try to reduce distance between inputs of same class and increase distances between inputs from different classes\n",
        "4. Siamese models are used when we have large number of classes say 100s or 1000s. The idea is to learn between 2 inputs.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y4Pjt48mRJ20"
      },
      "source": [
        "'''\n",
        "Model we choose here can be any model, CNN/RNN/ANN or any \n",
        "ML(decision trees/randomforest etc.) arcitecture.\n",
        "I have used an ANN with embedding as I was working on text data.\n",
        "\n",
        "Embedding layer works as a word2vec layer and converts words from our input \n",
        "sentence to a vector. Here, in the e.g. below, I have sentences with a fix\n",
        "length of 15 (after using padding). For every word in the sentence, it will\n",
        "create a vector of size 32.\n",
        "  Say we have 100 training examples, our input will be (100,15) i.e. 100 \n",
        "sentences of length 15 each. Out of the embedding layer will be (100,15,32) i.e.\n",
        "100 vectors with 15 words each and each word is represented by a vector of\n",
        "length 32\n",
        "'''\n",
        "\n",
        "vocab = 100\n",
        "model = Sequential()\n",
        "model.add(Embedding(input_dim=vocab, output_dim= 32, input_length=15))\n",
        "model.add(GlobalAveragePooling1D())\n",
        "model.add(Dense(32, 'relu'))\n",
        "model.add(Dense(16, activation='relu'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8A6WtkRPRVXo",
        "outputId": "fbca031d-020c-4e0b-924c-ee65bc2b5cc1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 308
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 15, 32)            3200      \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 32)                1056      \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 16)                528       \n",
            "=================================================================\n",
            "Total params: 4,784\n",
            "Trainable params: 4,784\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J_EjJAdERVoE"
      },
      "source": [
        "'''\n",
        "Here we define the input shape as we have defined in the model, basically we \n",
        "define 2 tensors of the input shape same as our model's input shape\n",
        "'''\n",
        "left_input = Input(shape=(15))\n",
        "right_input = Input(shape=(15))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4o-kMOOmOLyh"
      },
      "source": [
        "'''\n",
        "We pass both our inputs from the same model and get the output embedding. Since,\n",
        "we had 100 examples, final output will be of shape (100,16) 16 is because of the\n",
        "last dense layer in the model\n",
        "'''\n",
        "left_input_embedding = model(left_input)\n",
        "right_input_embedding = model(right_input)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "knwXZ4UGOQZk"
      },
      "source": [
        "'''\n",
        "Keras provides us with a customizable layer called lambda where we can pass our\n",
        "function and get the output. Here, we want to calculate the distance between\n",
        "embeddings of 2 inputs. We pass the previously defined euclidean_distance\n",
        "function and pass the embeddings of 2 inputs\n",
        "'''\n",
        "\n",
        "lambdaLayer = Lambda(euclidean_distance,\n",
        "                  output_shape=eucl_dist_output_shape)([left_input_embedding, right_input_embedding])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wVOIbq8-OSeI"
      },
      "source": [
        "'''\n",
        "Model() method stitches the customized layers to the previously defined\n",
        "sequential layers.\n",
        "'''\n",
        "siamese_model = Model(inputs=[left_input, right_input], outputs = lambdaLayer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sig43W96OYnw",
        "outputId": "0ddf4edd-6509-4b50-c62f-9c9b930b34e0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "source": [
        "siamese_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"functional_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_2 (InputLayer)            [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "sequential (Sequential)         (None, 16)           4784        input_1[0][0]                    \n",
            "                                                                 input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lambda (Lambda)                 (None, 1)            0           sequential[0][0]                 \n",
            "                                                                 sequential[1][0]                 \n",
            "==================================================================================================\n",
            "Total params: 4,784\n",
            "Trainable params: 4,784\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YoRxdlljOXTx"
      },
      "source": [
        "'''\n",
        "Finally using a distance based loss function we compile and fit our model\n",
        "'''\n",
        "siamese_model.compile(loss=contrastive_loss, optimizer='adam', metrics=[accuracy])\n",
        "siamese_model.fit([data_1, data_2], y_data, batch_size=64,epochs=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f7A8Ef0bC_vp"
      },
      "source": [
        "'''\n",
        "Whole idea of a Siamese network is to learn a similarity function, it's either \n",
        "using contrastive loss of triplet loss etc. \n",
        "I noticed in some application people have added a sigmoid or softmax layer after\n",
        "the lambda layer and have tried to predict the class of interest. This is another\n",
        "application of siamese model but it takes the essence away as we use it when we\n",
        "have lot of classes and few example per class\n",
        "'''\n",
        "\n",
        "lambdaLayer = Dense(1, activation='sigmoid')(lambdaLayer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-XE36-OOEsxx"
      },
      "source": [
        "siamese_model = Model(inputs=[left_input, right_input], outputs = lambdaLayer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W1MlgSjrEhXW",
        "outputId": "cf3bde7c-8e25-430b-b4a6-d85e88c7414d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 381
        }
      },
      "source": [
        "siamese_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"functional_3\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_2 (InputLayer)            [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "sequential (Sequential)         (None, 16)           4784        input_1[0][0]                    \n",
            "                                                                 input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lambda (Lambda)                 (None, 1)            0           sequential[0][0]                 \n",
            "                                                                 sequential[1][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "dense_2 (Dense)                 (None, 1)            2           lambda[0][0]                     \n",
            "==================================================================================================\n",
            "Total params: 4,786\n",
            "Trainable params: 4,786\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nruswScBEe7v"
      },
      "source": [
        "'''\n",
        "Since we have an output layer with sigmoid activation, we can have the loss as \n",
        "binary crossentropy. Similarly, if we want to use softmax layer, we can use\n",
        "categorical crossentropy.\n",
        "'''\n",
        "\n",
        "siamese_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[accuracy])\n",
        "siamese_model.fit([data_1, data_2], y_data, batch_size=64,epochs=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cdCbM8A9VMI8"
      },
      "source": [
        "### **Training Different Networks for 2 Inputs**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jV761u4Xai4Z"
      },
      "source": [
        "Below is a variation of Siamese model, earlier we had 2 similar inputs and we\n",
        "passed them from 1 model to get embeddings. Suppose we have 2 totally different inputs, say one is image of a person and other is eye color or hair color etc.\n",
        "  In this case we need 2 separate models to spit out 2 different embedding and\n",
        "we can again use our distance based loss function to make the 2 embedding as\n",
        "close/far from each other depending on the case in hand.\n",
        "\n",
        "1.   We define model architecture 1\n",
        "2.   We define model architecture 2\n",
        "3. We pass the o/p of each of the modelsfrom a customized layer to get the distances\n",
        "4. We calculate the loss using the distance based contrastive or triplet loss functions\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EpmMqU3DVMI-"
      },
      "source": [
        "# Define Model Architecture 1\n",
        "\n",
        "def model_arc_1():\n",
        "\n",
        "  input = Input(batch_shape=(None,15,))\n",
        "\n",
        "  model_desc = Embedding(input_dim=100, output_dim= 32, input_length=15)(input)\n",
        "  model_desc = GlobalAveragePooling1D()(model_desc)\n",
        "  model_desc = Dense(32, 'relu')(model_desc)\n",
        "\n",
        "  return Model(input, model_desc)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nbvu9vWuVMJB"
      },
      "source": [
        "# Define Model Architecture 2\n",
        "\n",
        "def model_arc_2():\n",
        "\n",
        "  input = Input(batch_shape=(None,3,))\n",
        "\n",
        "  model_icd = Embedding(input_dim=100, output_dim= 32, input_length=3)(input)\n",
        "  model_icd = GlobalAveragePooling1D()(model_icd)\n",
        "  model_icd = Dense(32, 'relu')(model_icd)\n",
        "\n",
        "  return Model(input, model_icd)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Dt5z8VTVMJF"
      },
      "source": [
        "# Create objects of both model types\n",
        "model1 = model_arc_1()\n",
        "model2 = model_arc_2()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ko7MdTfkVMJI"
      },
      "source": [
        "#Customized layer for calculating distances\n",
        "lambdaLayer = Lambda(euclidean_distance,\n",
        "                  output_shape=eucl_dist_output_shape)([model1.output, model2.output])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f_T_b-UQVMJL"
      },
      "source": [
        "#Stitch the layers using Model function and define\n",
        "siamese_model_v2 = Model(inputs=[model1.input, model2.input], outputs = lambdaLayer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IZbtnj3OeVo_",
        "outputId": "1e485a7e-6f63-4783-fdf8-163467c16533",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "siamese_model_v2.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"functional_19\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_17 (InputLayer)           [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_18 (InputLayer)           [(None, 3)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_6 (Embedding)         (None, 15, 32)       3200        input_17[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "embedding_7 (Embedding)         (None, 3, 32)        3200        input_18[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "global_average_pooling1d_6 (Glo (None, 32)           0           embedding_6[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "global_average_pooling1d_7 (Glo (None, 32)           0           embedding_7[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "dense_9 (Dense)                 (None, 32)           1056        global_average_pooling1d_6[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "dense_10 (Dense)                (None, 32)           1056        global_average_pooling1d_7[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "lambda_3 (Lambda)               (None, 1)            0           dense_9[0][0]                    \n",
            "                                                                 dense_10[0][0]                   \n",
            "==================================================================================================\n",
            "Total params: 8,512\n",
            "Trainable params: 8,512\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "86iA6NhaVMJN"
      },
      "source": [
        "siamese_model.compile(loss=contrastive_loss, optimizer='adam', metrics=[accuracy])\n",
        "siamese_model.fit([tokenized_data_1, tokenized_data_2], y_data, batch_size=64,epochs=1)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}