dreambooth_runpod_vastai_joepenna_obsolete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aa2c1ada",
   "metadata": {
    "id": "aa2c1ada"
   },
   "source": [
    "# Dreambooth\n",
    "### Notebook implementation by Joe Penna (@MysteryGuitarM on Twitter) - Improvements by David Bielejeski\n",
    "\n",
    "### Instructions\n",
    "- Sign up for RunPod here: https://runpod.io/?ref=n8yfwyum\n",
    "    - Note: That's my personal referral link. Please don't use it if we are mortal enemies.\n",
    "\n",
    "- Click *Deploy* on either `SECURE CLOUD` or `COMMUNITY CLOUD`\n",
    "\n",
    "- Follow the rest of the instructions in this video: https://www.youtube.com/watch?v=7m__xadX0z0#t=5m33.1s\n",
    "\n",
    "Latest information on:\n",
    "https://github.com/JoePenna/Dreambooth-Stable-Diffusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b971cc0",
   "metadata": {
    "id": "7b971cc0"
   },
   "source": [
    "## Build Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2AsGA1xpNQnb",
   "metadata": {
    "id": "2AsGA1xpNQnb"
   },
   "outputs": [],
   "source": [
    "# If running on Vast.AI, copy the code in this cell into a new notebook. Run it, then launch the `dreambooth_runpod_joepenna.ipynb` notebook from the jupyter interface.\n",
    "!git clone https://github.com/JoePenna/Dreambooth-Stable-Diffusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# BUILD ENV\n",
    "!pip install numpy==1.23.1\n",
    "!pip install pytorch-lightning==1.7.6\n",
    "!pip install csv-logger\n",
    "!pip install torchmetrics==0.11.1\n",
    "!pip install torch-fidelity==0.3.0\n",
    "!pip install albumentations==1.1.0\n",
    "!pip install opencv-python==4.7.0.72\n",
    "!pip install pudb==2019.2\n",
    "!pip install omegaconf==2.1.1\n",
    "!pip install pillow==9.4.0\n",
    "!pip install einops==0.4.1\n",
    "!pip install transformers==4.25.1\n",
    "!pip install kornia==0.6.7\n",
    "!pip install diffusers[training]==0.3.0\n",
    "!pip install captionizer==1.0.1\n",
    "!pip install -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers\n",
    "!pip install -e git+https://github.com/openai/CLIP.git@main#egg=clip\n",
    "!pip install -e .\n",
    "!pip install huggingface_hub\n",
    "!pip install gitpython\n"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Download the 1.5 model from hugging face\n",
    "You can also provide your own v1.* model for training by uploading it and renaming it to \"model.ckpt\".  It should be in the same directory as dreambooth_runpod_joepenna.ipynb"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Download the 1.5 sd model\n",
    "from IPython.display import clear_output\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "downloaded_model_path = hf_hub_download(\n",
    " repo_id=\"panopstor/EveryDream\",\n",
    " filename=\"sd_v1-5_vae.ckpt\"\n",
    ")\n",
    "\n",
    "# Move the sd_v1-5_vae.ckpt to the root of this directory as \"model.ckpt\"\n",
    "actual_locations_of_model_blob = !readlink -f {downloaded_model_path}\n",
    "!mv {actual_locations_of_model_blob[-1]} model.ckpt\n",
    "clear_output()\n",
    "print(\"✅ model.ckpt successfully downloaded\")"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "markdown",
   "id": "17d1d11a",
   "metadata": {
    "id": "17d1d11a"
   },
   "source": [
    "# Regularization Images (Skip this section if you are uploading your own or using the provided images)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed07a5df",
   "metadata": {
    "id": "ed07a5df"
   },
   "source": [
    "Training teaches your new model both your token **but** re-trains your class simultaneously.\n",
    "\n",
    "From cursory testing, it does not seem like reg images affect the model too much. However, they do affect your class greatly, which will in turn affect your generations.\n",
    "\n",
    "You can either generate your images here, or use the repos below to quickly download 1500 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67f9ff0c-b529-4c7c-8e26-8388d70a5d91",
   "metadata": {
    "id": "67f9ff0c-b529-4c7c-8e26-8388d70a5d91"
   },
   "outputs": [],
   "source": [
    "# GENERATE 200 images - Optional\n",
    "self_generated_files_prompt = \"person\" #@param {type:\"string\"}\n",
    "self_generated_files_count = 200 #@param {type:\"integer\"}\n",
    "\n",
    "!python scripts/stable_txt2img.py \\\n",
    " --seed 10 \\\n",
    " --ddim_eta 0.0 \\\n",
    " --n_samples 1 \\\n",
    " --n_iter {self_generated_files_count} \\\n",
    " --scale 10.0 \\\n",
    " --ddim_steps 50 \\\n",
    " --ckpt model.ckpt \\\n",
    " --prompt {self_generated_files_prompt}\n",
    "\n",
    "dataset=self_generated_files_prompt\n",
    "\n",
    "!mkdir -p regularization_images/{dataset}\n",
    "!mv outputs/txt2img-samples/*.png regularization_images/{dataset}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d1c7e1c",
   "metadata": {
    "id": "3d1c7e1c"
   },
   "outputs": [],
   "source": [
    "# Zip up the files for downloading and reuse.\n",
    "# Download this file locally so you can reuse during another training on this dataset\n",
    "!apt-get install -y zip\n",
    "!zip -r regularization_images.zip regularization_images/{dataset}"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Download pre-generated regularization images\n",
    "We've created the following image sets\n",
    "\n",
    "* `man_euler` - provided by Niko Pueringer (Corridor Digital) - euler @ 40 steps, CFG 7.5\n",
    "* `man_unsplash` - pictures from various photographers\n",
    "* `person_ddim` - provided by David Bielejeski - ddim @ 50 steps, CFG 10.0\n",
    "* `woman_ddim` - provided by David Bielejeski - ddim @ 50 steps, CFG 10.0\n",
    "* `artstyle` - provided by Hackmans - ddim @ 50 steps, CFG 10.0\n",
    "\n",
    "`person_ddim` is recommended for training people\n"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7EydXCjOV1v",
   "metadata": {
    "id": "e7EydXCjOV1v"
   },
   "outputs": [],
   "source": [
    "#Download Regularization Images\n",
    "\n",
    "dataset=\"person_ddim\" #@param [\"man_euler\", \"man_unsplash\", \"person_ddim\", \"woman_ddim\", \"artstyle\"]\n",
    "!git clone https://github.com/djbielejeski/Stable-Diffusion-Regularization-Images-{dataset}.git\n",
    "\n",
    "!mkdir -p regularization_images/{dataset}\n",
    "!mv -v Stable-Diffusion-Regularization-Images-{dataset}/{dataset}/*.* regularization_images/{dataset}"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Upload your training images\n",
    "Upload 10-20 images of someone to\n",
    "\n",
    "```\n",
    "/workspace/Dreambooth-Stable-Diffusion/training_images\n",
    "```\n",
    "\n",
    "WARNING: Be sure to upload an *even* amount of images, otherwise the training inexplicably stops at 1500 steps.\n",
    "\n",
    "*   2-3 full body\n",
    "*   3-5 upper body\n",
    "*   5-12 close-up on face\n",
    "\n",
    "The images should be:\n",
    "\n",
    "- as close as possible to the kind of images you're trying to make"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "#@markdown Add here the URLs to the images of the subject you are adding\n",
    "urls = [\n",
    " \"https://i.imgur.com/test1.png\",\n",
    " \"https://i.imgur.com/test2.png\",\n",
    " \"https://i.imgur.com/test3.png\",\n",
    " \"https://i.imgur.com/test4.png\",\n",
    " \"https://i.imgur.com/test5.png\",\n",
    " # You can add additional images here -- about 20-30 images in different\n",
    "]"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#@title Download and check the images you have just added\n",
    "import os\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "def image_grid(imgs, rows, cols):\n",
    " assert len(imgs) == rows*cols\n",
    "\n",
    " w, h = imgs[0].size\n",
    " grid = Image.new('RGB', size=(cols*w, rows*h))\n",
    " grid_w, grid_h = grid.size\n",
    "\n",
    " for i, img in enumerate(imgs):\n",
    "  grid.paste(img, box=(i%cols*w, i//cols*h))\n",
    " return grid\n",
    "\n",
    "def download_image(url):\n",
    " try:\n",
    "  response = requests.get(url)\n",
    " except:\n",
    "  return None\n",
    " return Image.open(BytesIO(response.content)).convert(\"RGB\")\n",
    "\n",
    "images = list(filter(None,[download_image(url) for url in urls]))\n",
    "save_path = \"./training_images\"\n",
    "if not os.path.exists(save_path):\n",
    " os.mkdir(save_path)\n",
    "[image.save(f\"{save_path}/{i}.png\", format=\"png\") for i, image in enumerate(images)]\n",
    "image_grid(images, 1, len(images))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad4e50df",
   "metadata": {
    "id": "ad4e50df"
   },
   "source": [
    "## Training\n",
    "\n",
    "If training a person or subject, keep an eye on your project's `logs/{folder}/images/train/samples_scaled_gs-00xxxx` generations.\n",
    "\n",
    "If training a style, keep an eye on your project's `logs/{folder}/images/train/samples_gs-00xxxx` generations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fa5dd66-2ca0-4819-907e-802e25583ae6",
   "metadata": {
    "id": "6fa5dd66-2ca0-4819-907e-802e25583ae6",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Training\n",
    "\n",
    "# This isn't used for training, just to help you remember what your trained into the model.\n",
    "project_name = \"project_name\"\n",
    "\n",
    "# MAX STEPS\n",
    "# How many steps do you want to train for?\n",
    "max_training_steps = 2000\n",
    "\n",
    "# Match class_word to the category of the regularization images you chose above.\n",
    "class_word = \"person\" # typical uses are \"man\", \"person\", \"woman\"\n",
    "\n",
    "# If you are training a person's face, set this to True\n",
    "i_am_training_a_persons_face = False\n",
    "\n",
    "flip_p_arg = 0.0 if i_am_training_a_persons_face else 0.5\n",
    "\n",
    "# This is the unique token you are incorporating into the stable diffusion model.\n",
    "token = \"firstNameLastName\"\n",
    "\n",
    "# 0 Saves the checkpoint when max_training_steps is reached.\n",
    "# 250 saves the checkpoint every 250 steps as well as when max_training_steps is reached.\n",
    "save_every_x_steps = 0\n",
    "\n",
    "reg_data_root = \"/workspace/Dreambooth-Stable-Diffusion/regularization_images/\" + dataset\n",
    "\n",
    "!rm -rf training_images/.ipynb_checkpoints\n",
    "!python \"main.py\" \\\n",
    " --project_name \"{project_name}\" \\\n",
    " --debug False \\\n",
    " --max_training_steps {max_training_steps} \\\n",
    " --token \"{token}\" \\\n",
    " --training_model \"model.ckpt\" \\\n",
    " --training_images \"/workspace/Dreambooth-Stable-Diffusion/training_images\" \\\n",
    " --regularization_images \"{reg_data_root}\" \\\n",
    " --class_word \"{class_word}\" \\\n",
    " --flip_p {flip_p_arg} \\\n",
    " --save_every_x_steps {save_every_x_steps}"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Optional - Upload to google drive\n",
    "* run the following commands in a new `terminal` in the `Dreambooth-Stable-Diffusion` directory\n",
    "* `chmod +x ./gdrive`\n",
    "* `./gdrive about`\n",
    "* `paste your token here after navigating to the link`\n",
    "* `./gdrive upload trained_models/{file_name.ckpt}`"
   ],
   "metadata": {
    "collapsed": true
   }
  },
  {
   "cell_type": "markdown",
   "id": "9a90ac5c",
   "metadata": {},
   "source": [
    "# Big Important Note!\n",
    "\n",
    "The way to use your token is `<token> <class>` ie `joepenna person` and not just `joepenna`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d28d0139",
   "metadata": {},
   "source": [
    "## Generate Images With Your Trained Model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80ddb03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python scripts/stable_txt2img.py \\\n",
    " --ddim_eta 0.0 \\\n",
    " --n_samples 1 \\\n",
    " --n_iter 4 \\\n",
    " --scale 7.0 \\\n",
    " --ddim_steps 50 \\\n",
    " --ckpt \"/workspace/Dreambooth-Stable-Diffusion/trained_models/{file_name}\" \\\n",
    " --prompt \"joepenna person as a masterpiece portrait painting by John Singer Sargent in the style of Rembrandt\""
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}