Convert model weights from InstructDiffusion failed

[liming-ai]

Hi @yuzhou914 @yxgeee ,

Thanks a lot for your work, I tried to reproduce the training progress but always failed when I tried to convert the InstructDiffusion weights to diffusers as the instruction in the README:

/home/code/SmartEdit/convert_from_ckpt.py:57: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
  from diffusers.pipeline_utils import DiffusionPipeline
Traceback (most recent call last):
  File "/home/liming/code/SmartEdit/convert_original_stable_diffusion_to_diffusers.py", line 157, in <module>
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/home/liming/code/SmartEdit/convert_from_ckpt.py", line 1401, in download_from_original_stable_diffusion_ckpt
    set_module_tensor_to_device(unet, param_name, "cpu", value=param)
  File "/home/tiger/miniconda3/envs/smartedit/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 286, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([320, 4, 3, 3]) in "weight" (which has shape torch.Size([320, 8, 3, 3])), this looks incorrect.

Could you please check this procedure and give me some instructions? Thanks again.