ai-gateway/set-up-fallback-from-stable-diffusion-to-dall-e.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Portkey-AI/portkey-cookbook/blob/main/ai-gateway/set-up-fallback-from-stable-diffusion-to-dall-e.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4586cc53",
      "metadata": {
        "id": "4586cc53"
      },
      "source": [
        "# Set up Fallback from Stable Diffusion to Dall-E\n",
        "\n",
        "Generative AI models have revolutionized text generation and opened up new possibilities for developers. What next? A new category of image generation models.\n",
        "\n",
        "This cookbook introduces Portkey’s multimodal AI gateway, which helps you switch between multiple image generation models without any code changes — all with OpenAI SDK. You will learn to set up fallbacks from Stable Diffusion to Dall-E."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "42ae2283",
      "metadata": {
        "id": "42ae2283"
      },
      "source": [
        "## 1.  Integrate Image Gen Models with Portkey  \n",
        "\n",
        "Begin by storing API keys in the Portkey Vault.\n",
        "\n",
        "To save your OpenAI and StabilityAI keys in the Portkey Vault:\n",
        "1. Go to **portkey.ai**\n",
        "2. Click **Virtual Keys** and then **Create**\n",
        "    1. Enter **Name** and **API Key**,\n",
        "    2. Hit **Create**\n",
        "3. Copy the virtual key from the **KEY** column\n",
        "\n",
        "We successfully have set up virtual keys!\n",
        "\n",
        "For more information, refer the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys).\n",
        "\n",
        "The multi-modal AI gateway will use these virtual keys in the future to apply a fallback mechanism to every request from your app."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5e2efe10",
      "metadata": {
        "id": "5e2efe10"
      },
      "source": [
        "## 2. Making a call to Stability AI using OpenAI SDK\n",
        "\n",
        "With Portkey, you can call Stability AI models like SDXL right from inside the OpenAI SDK. Just change the `base_url` to Portkey Gateway and add `defaultHeaders` while instantiating your OpenAI client, and you're good to go\n",
        "\n",
        "Import the `openai` and `portkey_ai` libraries to send the requests, whereas the rest of the utility libraries will help decode the base64 response and print them onto Jupyter Notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9c6ac821",
      "metadata": {
        "id": "9c6ac821"
      },
      "outputs": [],
      "source": [
        "from IPython.display import display\n",
        "from PIL import Image\n",
        "from openai import OpenAI\n",
        "from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
        "import requests, io, base64, json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e6f4c81a",
      "metadata": {
        "id": "e6f4c81a"
      },
      "outputs": [],
      "source": [
        "PORTKEY_API_KEY=\"YOUR_PORTKEY_API_KEY_HERE\"\n",
        "OPENAI_VIRTUAL_KEY=\"YOUR_OPENAI_VIRTUAL_KEY_HERE\"\n",
        "CONFIG_ID=\"YOUR_CONFIG_ID_HERE\"\n",
        "OPENAI_API_KEY=\"REDUNDANT\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "96aa498e",
      "metadata": {
        "id": "96aa498e"
      },
      "source": [
        "Declare the arguments to pass to the parameters of OpenAI SDK and initialize a client instance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6383332c",
      "metadata": {
        "id": "6383332c"
      },
      "outputs": [],
      "source": [
        "STABILITYAI_VIRTUAL_KEY=\"YOUR_STABILITYAI_VIRTUAL_KEY_HERE\"\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key=\"REDUNDANT\",\n",
        "    base_url=PORTKEY_GATEWAY_URL,\n",
        "    default_headers=createHeaders(\n",
        "        provider=\"stabilityai\",\n",
        "        api_key=PORTKEY_API_KEY,\n",
        "        virtual_key=STABILITYAI_VIRTUAL_KEY,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "75e47fca",
      "metadata": {
        "id": "75e47fca"
      },
      "source": [
        "The `api_key` parameter is passed a random string since it’s redundant as the request will be handled through Portkey.\n",
        "\n",
        "To generate an image:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e15ab799",
      "metadata": {
        "scrolled": true,
        "id": "e15ab799"
      },
      "outputs": [],
      "source": [
        "image = client.images.generate(\n",
        "  model=\"stable-diffusion-v1-6\",\n",
        "  prompt=\"Kraken in the milkyway galaxy\",\n",
        "  n=1,\n",
        "  size=\"1024x1024\",\n",
        "  response_format=\"b64_json\"\n",
        ")\n",
        "\n",
        "base64_image = image.data[0].b64_json\n",
        "\n",
        "image_data = base64.b64decode(base64_image)\n",
        "\n",
        "image = Image.open(io.BytesIO(image_data))\n",
        "display(image)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "900e40aa",
      "metadata": {
        "id": "900e40aa"
      },
      "source": [
        "The image you receive in the response is encoded in base64 format, which requires you to decode it before you can view it in the Jupyter Notebook. In addition, Portkey offers logging for observability. To find all the information for every request, simply check the requests on the **Dashboard > Logs**."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "55edb172",
      "metadata": {
        "id": "55edb172"
      },
      "source": [
        "## 3. Now, Setup a Fallback from SDXL to Dall-E\n",
        "\n",
        "Let’s learn how to enhance the reliability of your Stability AI requests by configuring automatic fallbacks to Dall-E in case of failures. You can use Gateway Configs on Portkey to implement this automated fallback logic. These configurations can be passed while creating your OpenAI client.\n",
        "\n",
        "From the Portkey Dashboard, open **Configs** and then click **Create**. In the config editor, write the JSON for Gateway Configs:\n",
        "\n",
        "```json\n",
        "{\n",
        "  \"strategy\": {\n",
        "    \"mode\": \"fallback\"\n",
        "  },\n",
        "  \"targets\": [\n",
        "    {\n",
        "      \"virtual_key\": \"stability-ai-virtualkey\",\n",
        "      \"override_params\": {\n",
        "        \"model\": \"stable-diffusion-v1-6\"\n",
        "      }\n",
        "    },\n",
        "    {\n",
        "      \"virtual_key\": \"open-ai-virtual-key\",\n",
        "      \"override_params\": {\n",
        "        \"model\": \"dall-e-3\"\n",
        "      }\n",
        "    }\n",
        "  ]\n",
        "}\n",
        "```\n",
        "\n",
        "These configs tell the AI gateway to follow an `fallback` strategy, where the primary target to forward requests to is _Stability AI_ (automatically inferred from the virtual key) and then to _OpenAI_. The `override_params` let’s you override the default models for the provider. Finally, surprise surprise! — we also enabled caching with just one more key-value pair.\n",
        "\n",
        "Learn about [Gateway Configs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs) and [Caching](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic) from the docs.\n",
        "\n",
        "Hit **Save Config** on the top right corner and grab the **Config ID. **Next up, we are going to use the _Config ID _in our requests to activate fallback mechanism."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0681664d",
      "metadata": {
        "id": "0681664d"
      },
      "source": [
        "## 4. Make a request with gateway configs  \n",
        "\n",
        "Finally, the requests will be sent like we did with OpenAI SDK earlier, but with one specific difference - the `config` parameter. The request is sent through Portkey and uses saved gateway configs as references to activate the fallback mechanism."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e34d9797",
      "metadata": {
        "id": "e34d9797"
      },
      "outputs": [],
      "source": [
        "client = OpenAI(\n",
        "    api_key=OPENAI_API_KEY,\n",
        "    base_url=PORTKEY_GATEWAY_URL,\n",
        "    default_headers=createHeaders(\n",
        "        api_key=PORTKEY_API_KEY,\n",
        "        config=CONFIG_ID\n",
        "    )\n",
        ")\n",
        "\n",
        "image = client.images.generate(\n",
        "  model=\"stable-diffusion-v1-6\",\n",
        "  prompt=\"Harry Potter travelling the world using Portkey\",\n",
        "  n=1,\n",
        "  size=\"1024x1024\",\n",
        "  response_format=\"b64_json\"\n",
        ")\n",
        "\n",
        "base64_image = image.data[0].b64_json\n",
        "\n",
        "image_data = base64.b64decode(base64_image)\n",
        "\n",
        "image = Image.open(io.BytesIO(image_data))\n",
        "display(image)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1053e49e",
      "metadata": {
        "id": "1053e49e"
      },
      "source": [
        "## Afterthoughts\n",
        "\n",
        "All the requests that go through Portkey will appear in the Logs page within the Portkey Dashboard. You can apply filters or even trace the specific set of requests. Check out [Request Tracing](https://portkey.ai/docs/product/observability-modern-monitoring-for-llms/traces). Simultaneously, a fallback icon is turned on for the log where the fallback is activated.\n",
        "\n",
        "Portkey supports multiple providers offering multimodal capabilities, such as OpenAI, Anthropic, and Stability AI, all accessible through a unified API interface following OpenAI Signature.\n",
        "\n",
        "For further exploration, why not [play with Vision capabilities](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/multimodal-capabilities/vision)?"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.18"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}