Coqui_Speech_to_Text_Test.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Building a Coqui Speech to Text Client",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Building a Coqui Speech to Text Client\n",
        "\n",
        "The Coqui project uses Bazel as its main build system but it's quite a complex and daunting tool for most users. Happily the team also provide binary releases of all the libraries, so this notebook shows how you can use the existing makefile to build the example `stt` client executable, using just the binary release without Bazel.\n",
        "The goal of this script is to demonstrate how you can easily integrate speech to text into your own programs."
      ],
      "metadata": {
        "id": "6UvpBAfRiYrD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download Binary Release\n",
        "\n",
        "First we download the example executable, `stt`, and the shared library, `libstt.so`, that contains the compiled framework code, all parts of the native_client package."
      ],
      "metadata": {
        "id": "a40bqVQ-jpn8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!mkdir -p stt_binary_release\n",
        "!wget --quiet https://github.com/coqui-ai/STT/releases/download/v1.1.0/native_client.tflite.Linux.tar.xz\n",
        "!unxz native_client.tflite.Linux.tar.xz\n",
        "!tar -xf native_client.tflite.Linux.tar --directory stt_binary_release"
      ],
      "metadata": {
        "id": "IfMgxEidf5ix"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download Source Code\n",
        "\n",
        "Here we're going to download the full source code package for STT, but we're only going to actually use a few files inside the native_client folder. I chose to go this route rather than forking the repo and cherry-picking just the files we need because this approach should be easier to keep in sync with code updates from Coqui."
      ],
      "metadata": {
        "id": "WMoTB5B5kk9i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget --quiet https://github.com/coqui-ai/STT/archive/refs/tags/v1.1.0.tar.gz\n",
        "!tar -xzf v1.1.0.tar.gz\n",
        "!mv STT-1.1.0 stt_source_code"
      ],
      "metadata": {
        "id": "moGNl69Rkkjg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Audio Example Files\n",
        "\n",
        "To demonstrate how the speech to text tool works, we need some WAV files to try it out on. Luckily Coqui provide some examples, together with transcripts of the expected output."
      ],
      "metadata": {
        "id": "g0ve2HSqnGNa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget --quiet https://github.com/coqui-ai/STT/releases/download/v1.1.0/audio-1.1.0.tar.gz\n",
        "!tar -xzf audio-1.1.0.tar.gz"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2GHmgUXvnF4l",
        "outputId": "949319fe-ad10-46f4-bf13-e3d437149536"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.macl'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.macl'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.macl'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.macl'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Recognition Model Download\n",
        "\n",
        "For this example I've chosen the English large vocabulary model, but there are over 80 different versions available for many languages at [coqui.ai/models](https://coqui.ai/models). Note that this is the *recognition* model, not the *language* model. Language models are used to post-process the results of the neural network, and are optional. To keep things simple, in this example we're just using the raw recognition model output, but there are lots of options to improve the quality for a particular application if you investigate things like language models and hotwords."
      ],
      "metadata": {
        "id": "Je4elXAwZUw6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget --quiet https://github.com/coqui-ai/STT-models/releases/download/english/coqui/v1.0.0-large-vocab/model.tflite"
      ],
      "metadata": {
        "id": "gV_dYKCAWbYt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## SOX Library Installation\n",
        "\n",
        "The example client program uses the SOX audio library to read WAV files and convert them to the expected sample rate. Since a Colab instance doesn't have SOX installed by default, we'll use `apt-get` to install it."
      ],
      "metadata": {
        "id": "D7x3XZWAZd8a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!apt-get -qq install -y libsox-dev"
      ],
      "metadata": {
        "id": "LGLl_BFjVg2Q",
        "outputId": "f5ed3f16-7cc1-4997-f7c5-3204118449d9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Selecting previously unselected package libopencore-amrnb0:amd64.\n",
            "(Reading database ... 155632 files and directories currently installed.)\n",
            "Preparing to unpack .../00-libopencore-amrnb0_0.1.3-2.1_amd64.deb ...\n",
            "Unpacking libopencore-amrnb0:amd64 (0.1.3-2.1) ...\n",
            "Selecting previously unselected package libopencore-amrwb0:amd64.\n",
            "Preparing to unpack .../01-libopencore-amrwb0_0.1.3-2.1_amd64.deb ...\n",
            "Unpacking libopencore-amrwb0:amd64 (0.1.3-2.1) ...\n",
            "Selecting previously unselected package libmagic-mgc.\n",
            "Preparing to unpack .../02-libmagic-mgc_1%3a5.32-2ubuntu0.4_amd64.deb ...\n",
            "Unpacking libmagic-mgc (1:5.32-2ubuntu0.4) ...\n",
            "Selecting previously unselected package libmagic1:amd64.\n",
            "Preparing to unpack .../03-libmagic1_1%3a5.32-2ubuntu0.4_amd64.deb ...\n",
            "Unpacking libmagic1:amd64 (1:5.32-2ubuntu0.4) ...\n",
            "Selecting previously unselected package libao-common.\n",
            "Preparing to unpack .../04-libao-common_1.2.2+20180113-1ubuntu1_all.deb ...\n",
            "Unpacking libao-common (1.2.2+20180113-1ubuntu1) ...\n",
            "Selecting previously unselected package libao4:amd64.\n",
            "Preparing to unpack .../05-libao4_1.2.2+20180113-1ubuntu1_amd64.deb ...\n",
            "Unpacking libao4:amd64 (1.2.2+20180113-1ubuntu1) ...\n",
            "Selecting previously unselected package libid3tag0:amd64.\n",
            "Preparing to unpack .../06-libid3tag0_0.15.1b-13_amd64.deb ...\n",
            "Unpacking libid3tag0:amd64 (0.15.1b-13) ...\n",
            "Selecting previously unselected package libmad0:amd64.\n",
            "Preparing to unpack .../07-libmad0_0.15.1b-9ubuntu18.04.1_amd64.deb ...\n",
            "Unpacking libmad0:amd64 (0.15.1b-9ubuntu18.04.1) ...\n",
            "Selecting previously unselected package libsox3:amd64.\n",
            "Preparing to unpack .../08-libsox3_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox3:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-alsa:amd64.\n",
            "Preparing to unpack .../09-libsox-fmt-alsa_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-alsa:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-ao:amd64.\n",
            "Preparing to unpack .../10-libsox-fmt-ao_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-ao:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-base:amd64.\n",
            "Preparing to unpack .../11-libsox-fmt-base_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-base:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-mp3:amd64.\n",
            "Preparing to unpack .../12-libsox-fmt-mp3_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-mp3:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-oss:amd64.\n",
            "Preparing to unpack .../13-libsox-fmt-oss_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-oss:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-pulse:amd64.\n",
            "Preparing to unpack .../14-libsox-fmt-pulse_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-pulse:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-fmt-all:amd64.\n",
            "Preparing to unpack .../15-libsox-fmt-all_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-fmt-all:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Selecting previously unselected package libsox-dev:amd64.\n",
            "Preparing to unpack .../16-libsox-dev_14.4.2-3ubuntu0.18.04.1_amd64.deb ...\n",
            "Unpacking libsox-dev:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libid3tag0:amd64 (0.15.1b-13) ...\n",
            "Setting up libao-common (1.2.2+20180113-1ubuntu1) ...\n",
            "Setting up libmagic-mgc (1:5.32-2ubuntu0.4) ...\n",
            "Setting up libmagic1:amd64 (1:5.32-2ubuntu0.4) ...\n",
            "Setting up libopencore-amrnb0:amd64 (0.1.3-2.1) ...\n",
            "Setting up libmad0:amd64 (0.15.1b-9ubuntu18.04.1) ...\n",
            "Setting up libopencore-amrwb0:amd64 (0.1.3-2.1) ...\n",
            "Setting up libsox3:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-fmt-mp3:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-fmt-base:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-fmt-pulse:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libao4:amd64 (1.2.2+20180113-1ubuntu1) ...\n",
            "Setting up libsox-fmt-alsa:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-fmt-oss:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-fmt-ao:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-fmt-all:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Setting up libsox-dev:amd64 (14.4.2-3ubuntu0.18.04.1) ...\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1.3) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Compiling the STT Client\n",
        "\n",
        "Now we have all the dependencies, we can invoke `make` to build the `stt` client program. Because the makefile expects to be run from inside the `native_client` folder, we use the `-C` option to direct `make` to the right path. The only change we have to apply to the standard makefile is overriding the expected location of the libraries to point to the folder we downloaded containing the binary releases, rather than the default of looking for them in a `Bazel` build folder.\n",
        "\n",
        "With v1.1.0 you'll see some log messages like `objdump: '/usr/lib/x86_64-linux-gnu/libmagic.so': No such file`, but these appear to be harmless warnings rather than true errors. This notebook's not official documentation, just an accompaniment to the blog post at [petewarden.com/2021/12/27/how-to-get-started-with-coquis-open-source-on-device-speech-to-text-tool/](https://petewarden.com/2021/12/27/how-to-get-started-with-coquis-open-source-on-device-speech-to-text-tool/).\n"
      ],
      "metadata": {
        "id": "yZKpHKgYnmyl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!make -C stt_source_code/native_client stt LINK_PATH_STT=-L../../stt_binary_release"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XhZ5kWjuUUx6",
        "outputId": "c48e4da9-a1c3-45da-ef5c-b678071de207"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "make: Entering directory '/content/stt_source_code/native_client'\n",
            "objdump: '/usr/lib/x86_64-linux-gnu/libmagic.so': No such file\n",
            "objdump: '/usr/lib/x86_64-linux-gnu/libmagic.so': No such file\n",
            "c++   -std=c++11 -o stt -I/content/stt_source_code/sox-build/include client.cc  -Wl,--no-as-needed -Wl,-rpath,\\$ORIGIN -L../../stt_binary_release  -lstt -lkenlm -ltflitedelegates -ltensorflowlite  -L/content/stt_source_code/sox-build/lib -lsox\n",
            "make: Leaving directory '/content/stt_source_code/native_client'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Running the Tool\n",
        "\n",
        "We should now have the `stt` example program built, so let's run it! For the purposes of keeping this notebook I've avoided installing the shared libraries from the binary in system paths, so we do have to tell the shell where to find our downloaded versions using the `LD_LIBRARY_PATH` environment variable. In typical deployment, these would have been copied to somewhere like `/usr/lib` during installation and so the variable wouldn't be needed.\n",
        "\n",
        "If everything's worked successfully, you should see some version information followed by a transcript of the audio file, \"why should one halt on the way\". Now you can start modifying the `stt_source_code/native_client/client.cc` source file to build your own program using speech to text. There's a GitHub repo showing a minimal set of files at [github.com/petewarden/stt_standalone_client](https://github.com/petewarden/stt_standalone_client)."
      ],
      "metadata": {
        "id": "zSwXiLnCb0A7"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qv5Xohcqe9Gu",
        "outputId": "8394ac37-180d-483e-f084-e353bb5c4654"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TensorFlow: v2.3.0-14-g4bdd3955115\n",
            " Coqui STT: v1.1.0-0-gf3605e23\n",
            "if everything works successfully you should see some vorshan in formation followed by a transcripe of dhogo pire why should one halt on doiy now your can start more difying the source wile the wild or own progrem using thes spitch to texts\n"
          ]
        }
      ],
      "source": [
        "!LD_LIBRARY_PATH=stt_binary_release stt_source_code/native_client/stt --model ./model.tflite --audio ./audio/Recording_1.wav"
      ]
    }
  ]
}