Skip to content

X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model

Notifications You must be signed in to change notification settings

Semolce9/X-Adapter

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

X-Adapter

This repository is the official implementation of X-Adapter.

X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, Mike Zheng Shou

Project Website arXiv

Overview_v7

X-Adapter enable plugins pretrained on old version (e.g. SD1.5) directly work with the upgraded Model (e.g., SDXL) without further retraining.

News

  • [17/02/2024] Inference code released

Setup

Requirements

conda create -n xadapter python=3.10
conda activate xadapter

pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

Installing xformers is highly recommended for high efficiency and low GPU cost.

Weights

[Stable Diffusion] Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input. The pre-trained Stable Diffusion models can be downloaded from Hugging Face (e.g., Stable Diffusion v1-5). You can also use fine-tuned Stable Diffusion models trained on different styles (e.g., Anything V4.0, Redshift, etc.).

[ControlNet] Controlnet is a method to control diffusion model with spatial conditions. You can download ControlNet family here.

[LoRA] LoRA is a lightweight adapter to fine-tune large-scale pretrained model. It is widely used for style or identity customization in diffusion model. You can download LoRA from diffusion community(e.g., civitai).

Checkpoint

Models can be downloaded from our HuggingFace page. Put the checkpoint in folder ./checkpoint/X-Adapter.

Usage

After preparing all checkpoints, we can run inference code using different plugins. You can refer to this tutorial to quickly get started with X-Adapter.

Controlnet Inference

Set --controlnet_canny_path or --controlnet_depth_path to ControlNet's path in the bash script. The default value is its huggingface model card.

sh ./bash_scripts/canny_controlnet_inference.sh
sh ./bash_scripts/depth_controlnet_inference.sh

LoRA Inference

Set --lora_model_path to LoRA's checkpoint in the bash script. In this example we use MoXin, and we put it in folder ./checkpoint/lora.

sh ./bash_scripts/lora_inference.sh

Controlnet-Tile Inference

Set --controlnet_tile_path to ControlNet-tile's path in the bash script. The default value is its huggingface model card.

sh ./bash_scripts/controlnet_tile_inference.sh

Cite

If you find X-Adapter useful for your research and applications, please cite us using this BibTeX:

@article{ran2023xadapter,
  title={X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model},
  author={Lingmin Ran and Xiaodong Cun and Jia-Wei Liu and Rui Zhao and Song Zijie and Xintao Wang and Jussi Keppo and Mike Zheng Shou},
  journal={arXiv preprint arXiv:2312.02238},
  year={2023}
}

About

X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 99.7%
  • Shell 0.3%