SDXL?

[NeusZimmer]

I saw SDXL to be ready to convert run on ONNX ,but with optimum, and with an script conversion for fp16 only working with CUDA, had someone already converted it? I would like to use it for a low mem AMD card....

[ssube]

If SDXL has been converted to ONNX, then I definitely want to add support for it. I have the SDXL weights downloaded and set up for CUDA, but I haven't had a chance to dig into the ONNX side. Do you have that script handy?

Getting the basic pipeline set up should not be too bad. It will take a little bit of work to get XL working with highres, because the current highres implementation runs txt2img followed by img2img in image space, where as XL is using latent space. That's something like encoder -> unet -> decoder -> encoder -> unet -> decoder vs encoder -> unet -> unet -> decoder, afaict.

I also want to allow passing numpy data between pipelines for other reasons and optimizations, so I'm curious how the results would look if we did SD v1.x highres in latent space instead, but I'll have to try that later.

[NeusZimmer]

HI, thanks for the answer...as said, i found a converter script, but only working with optimum, i was trying to avoid it, as it does not work good into AMD cards on windows...
Regarding hires & numpy data, I've being experimenting with re-use of the generated latent for some time, it ended into 2 approaches for ONNX, one experiment to create images based in a summatory of previous latents and one option for hi.res (txt2img + 1 to many img2img)., could check them in my UI approach (gradio based).... maybe want to check about it...
pipeline_onnx_stable_diffusion_hires_txt2img - toshare.txt

[NeusZimmer]

And, a lot of thanks for your works, it provide me with some ideas and workarounds...

[ssube]

That pipeline looks like what I was thinking and has some neat features, keeping track of the latents from each step. I will have to mess with it some more.

I've been working on a branch for SDXL and have made some progress, but I've been running into a lot of memory errors and tensor error that I am still tracking down (at the end). I had no problems converting the weights, but attempting to load them from FP16 on a 24GB CUDA card is failing, which is a little bit surprising since the model ended up being about 13GB. It should not be expanding that much after loading. I already have code in place to offload certain models to the CPU, so once I figure out why the initial memory usage is so high, selectively offloading some models should help. Using the XL pipelines from optimum seems to work for the most part and the code changes for txt2img and img2img are pretty minor. Getting XL hires working will be a little bit more work.

[ssube]

I have a branch working with the SDXL base model and some custom txt2img models. I was able to generate 512x512 with a fully fp32 model and 1024x1024 after converting the Unet to fp16, using a 16GB card.

Caveats: I haven't started testing on lower memory cards yet, I haven't tested the refiner or img2img yet, and I haven't tested any of this on the DirectML provider (Windows).

The changes so far are on the https://github.com/ssube/onnx-web/tree/feat/00-sdxl branch. I've tested it under ORT 1.15 and PyTorch 2.0, with both the CUDA and ROCm providers, and the base model is working reliably.

To convert SDXL models to ONNX from repositories on the HuggingFace hub:

optimum-cli export onnx --model stabilityai/stable-diffusion-xl-base-1.0 --task stable-diffusion-xl ../models/diffusion-sd_xl_onnx/

To convert SDXL models from .safetensors files from Civitai:

import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline

model = "/tmp/dreamshaper-xl"
pipe = StableDiffusionXLPipeline.from_single_file(f"{model}.safetensors", use_safetensors=True)
pipe.save_pretrained(model)

followed by

optimum-cli export onnx --model /tmp/dreamshaper-xl --task stable-diffusion-xl ../models/diffusion-dreamshaper-xl/

To optimize the Unet to fp16:

import onnx
import torch
import onnxruntime

from onnx.shape_inference import infer_shapes_path
from onnxruntime.transformers.float16 import convert_float_to_float16

model = "/opt/onnx-web/models/diffusion-dreamshaper-xl"
infer_shapes_path(f"{model}/unet/model.onnx")
unet = onnx.load(f"{model}/unet/model.onnx")
opt_model = convert_float_to_float16(unet, disable_shape_infer=True, force_fp16_initializers=True, keep_io_types=True, op_block_list=["Attention", "MultiHeadAttention"])
onnx.save_model(opt_model, f"{model}/unet/model-fp16.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="weights.pb")

A few examples:

[ssube]

I've merged the SDXL code for the existing pipelines (txt2img, img2img, and inpaint).

Converting modules is still a manual process, I need to figure that out next. There are also a few bugs that have come up in the optimum library:

• SDXL ORT base example produces broken images on Windows 10 huggingface/optimum#1310
• send both negative prompt embeds to ORT SDXL huggingface/optimum#1339
• one more branch I am working on, for a bug where the UNet can be loaded twice, causing an out-of-memory error