Expressions_Two_Ts.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Validação das Expressões Two-Ts.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/jcheong0428/c16146b386ea60fab888b56e8e5ee747/openface_shared.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hMhPBvnrARcd"
      },
      "source": [
        "# *Two-Ts*: Reproduction of Facial Expressions in Robotic Head with 3D Technology\n",
        "\n",
        "Master's degree in Computer Science at Federal University of ABC.\n",
        "\n",
        "Student: Tamires dos Santos.\n",
        "\n",
        "Advisor: Wagner Tanaka Botelho.\n",
        "\n",
        "Scholarships Received:\n",
        "+ Technical Training (TT-3) of the PIPE Program of the São Paulo Research Support Foundation, with process number 2019/08079-1, by Startup NTU Software Technology of the PIPE Phase 2 Project (Process Number: 2018/04306-0) from June/2019 to May/2020;\n",
        "+ Master's Scholarship from the Coordination for the Improvement of Higher Education Personnel from October/2020 to February/2021.\n",
        "\n",
        "\n",
        "\n",
        "Modification Date: July/2021."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XiVMHhCOMzyq"
      },
      "source": [
        "# Expression Validation System of the *Two-Ts*\n",
        "\n",
        "\n",
        "This notebook uses the open source project [OpenFace](https://github.com/TadasBaltrusaitis/OpenFace), by Tadas Baltrusaitis, to detect from a face, the FACS based Action Units (AUs) created by [Paul Ekman](https://www.paulekman.com/) in partnership with Wallace Friesen.\n",
        "\n",
        "In addition to the instructions on OpenFace's github, the article and notebook created by Jin Hyun Cheong (https://towardsdatascience.com/how-to-extract-facial-expressions-head-pose-and-gaze-from-any-youtube-video-2aa6590c2bb6).\n",
        "\n",
        "The purpose of the code is to perform the validation of facial expressions, during the reproduction of the six basic and universal emotions, on the face of the *Two-Ts* Virtual and Real, compared to Tamires dos Santos."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlvZkuHc0ynn"
      },
      "source": [
        "### Installing OpenFace"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BXBkyjI_GqtQ"
      },
      "source": [
        "import os\n",
        "from os.path import exists, join, basename, splitext\n",
        "\n",
        "################# Install CUDA 10.0 #################\n",
        "!apt-get --purge remove cuda nvidia* libnvidia-*\n",
        "!dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge\n",
        "!apt-get remove cuda-*\n",
        "!apt autoremove\n",
        "!apt-get update\n",
        "!wget  --no-clobber https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb\n",
        "!dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb\n",
        "!sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub\n",
        "!apt-get update\n",
        "!apt-get install cuda-10-0\n",
        "!wget --no-clobber http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb\n",
        "!apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb\n",
        "!apt-get update\n",
        "#######################################################\n",
        "\n",
        "################# Install OpenFace #################\n",
        "git_repo_url = 'https://github.com/TadasBaltrusaitis/OpenFace.git'\n",
        "project_name = splitext(basename(git_repo_url))[0]\n",
        "!git clone -q --depth 1 $git_repo_url\n",
        "#######################################################\n",
        "\n",
        "################# Install other packages #################\n",
        "!wget -q https://cmake.org/files/v3.13/cmake-3.13.0-Linux-x86_64.tar.gz\n",
        "!tar xfz cmake-3.13.0-Linux-x86_64.tar.gz --strip-components=1 -C /usr/local\n",
        "\n",
        "!sudo apt-get update\n",
        "!sudo apt-get install build-essential \n",
        "!sudo apt-get install g++-8\n",
        "#######################################################\n",
        "\n",
        "# Finishing the installation\n",
        "!cd OpenFace && bash ./download_models.sh && sudo bash ./install.sh\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8QW173G15ev"
      },
      "source": [
        "### Select images \n",
        "\n",
        "Find the images and extract the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QdVSEBT8XdVn"
      },
      "source": [
        "# Create folder to store data\n",
        "!rm -rf processed\n",
        "\n",
        "!./OpenFace/build/bin/FaceLandmarkImg -f virtual.jpg -out_dir processed -of dataImage1   #Two-Ts Virtual\n",
        "!./OpenFace/build/bin/FaceLandmarkImg -f real.jpg -out_dir processed -of dataImage2  #Two-Ts Real\n",
        "!./OpenFace/build/bin/FaceLandmarkImg -f tamires.png -out_dir processed -of dataImage3  #Tamires"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Z1nUfT22bMH"
      },
      "source": [
        "###Load the data for each image\n",
        "\n",
        "The image data will be loaded into Pandas, renaming the columns to eliminate empty spaces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ax8JJ3ayfIyO"
      },
      "source": [
        "import pandas as pd, seaborn as sns\n",
        "import matplotlib.pyplot as plt \n",
        "\n",
        "df1 = pd.read_csv('processed/dataImage1.csv')\n",
        "df1.columns = [col.replace(\" \", \"\") for col in df1.columns]\n",
        "\n",
        "df2 = pd.read_csv('processed/dataImage2.csv')\n",
        "df2.columns = [col.replace(\" \", \"\") for col in df2.columns]\n",
        "\n",
        "df3 = pd.read_csv('processed/dataImage3.csv')\n",
        "df3.columns = [col.replace(\" \", \"\") for col in df3.columns]\n",
        "\n",
        "print(df1)\n",
        "print(df2)\n",
        "print(df3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPvbE8Tb3Dqv"
      },
      "source": [
        "### Plot the results of the AUs\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aXD88gVKCc85"
      },
      "source": [
        "import re\n",
        "import numpy as np\n",
        "\n",
        "# Function to read the column values of the AUs\n",
        "def valuesAUs(id_image):\n",
        "  au_values = []\n",
        "  au_list = ['01', '02', '04', '05', '06' ,'07', '09', '10', '12', '14', '15', '17', '20', '23', '25', '26' , '45']\n",
        "  \n",
        "  for i in range(0, len(au_list)):\n",
        "    x = 'AU'+ au_list[i] +\"_r\"\n",
        "    if id_image == 1:\n",
        "        au_values.append(df1[x][0])\n",
        "    if id_image == 2:\n",
        "      au_values.append(df2[x][0])\n",
        "    if id_image == 3:\n",
        "      au_values.append(df3[x][0])\n",
        "  \n",
        "  return au_values\n",
        "\n",
        "listName = ['AU1', 'AU2', 'AU4', 'AU5', 'AU6' ,'AU7', 'AU9', 'AU10', 'AU12', 'AU14', 'AU15', 'AU17', 'AU20', 'AU23', 'AU25', 'AU26' , 'AU45']\n",
        "\n",
        "#au_regex_pat = re.compile(r'^AU[0-9]+_r$')\n",
        "#au_columns1 = df1.columns[df1.columns.str.contains(au_regex_pat)]\n",
        "\n",
        "au_values1 = valuesAUs(1)\n",
        "au_values2 = valuesAUs(2)\n",
        "au_values3 = valuesAUs(3)\n",
        "\n",
        "image1 =  np.arange(len(au_values1))\n",
        "image2 = [i + 0.2 for i in image1]\n",
        "image3 = [i + 0.4 for i in image1]\n",
        "\n",
        "plt.bar(image1, au_values1, width=0.2, label = 'Two-Ts Virtual', edgecolor ='g', color = 'white')\n",
        "plt.bar(image2, au_values2, width=0.2, label = 'Two-Ts Real', color = 'gray')\n",
        "plt.bar(image3, au_values3, width=0.2, label = 'Tamires', color = 'black')\n",
        "plt.xticks([i for i in range(len(au_values1))], listName)\n",
        "plt.xticks(rotation=20)\n",
        "plt.legend()\n",
        "plt.gcf().set_size_inches(20, 5) \n",
        "plt.ylabel('Intensity')\n",
        "plt.xlabel('Action Unit')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}