full funetuning sdxl RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x1280 and 2048x640)

[mobilejammer]

follow sd3.md my config.json as follow:
"--model_type": "full",
"--pretrained_model_name_or_path": "/home/ubuntu/stable-diffusion-xl-base-1.0",
"--model_family": "sdxl",

deepspeed config as follow:
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
gradient_accumulation_steps: 1
gradient_clipping: 1.0
offload_optimizer_device: none
offload_param_device: none
zero3_init_flag: false
zero3_save_16bit_model: false
zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
dynamo_config:
dynamo_backend: INDUCTOR
dynamo_mode: default
dynamo_use_dynamic: false
dynamo_use_fullgraph: false
enable_cpu_affinity: false
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

the error is:

detail is:
mat1 and mat2 shapes cannot be multiplied (77x1280 and 2048x640)
Traceback (most recent call last):
File "/home/ubuntu/SimpleTuner/train.py", line 71, in
trainer.train()
File "/home/ubuntu/SimpleTuner/helpers/training/trainer.py", line 2642, in train
model_pred = self.model_predict(
File "/home/ubuntu/SimpleTuner/helpers/training/trainer.py", line 2240, in model_predict
model_pred = self.unet(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1914, in forward
loss = self.module(*inputs, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
return inner()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in inner
result = forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/unets/unet_2d_condition.py", line 1216, in forward
sample, res_samples = downsample_block(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
return inner()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in inner
result = forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 1324, in forward
hidden_states = attn(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
return inner()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in inner
result = forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/transformers/transformer_2d.py", line 430, in forward
hidden_states = torch.utils.checkpoint.checkpoint(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/_compile.py", line 32, in inner
return disable_fn(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
return fn(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint
ret = function(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/transformers/transformer_2d.py", line 425, in custom_forward
return module(*inputs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
return inner()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in inner
result = forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/attention.py", line 552, in forward
attn_output = self.attn2(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
return inner()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in inner
result = forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 588, in forward
return self.processor(
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 3268, in call
key = attn.to_k(encoder_hidden_states)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1844, in _call_impl
return inner()
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in inner
result = forward_call(*args, **kwargs)
File "/home/ubuntu/SimpleTuner/.venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
return F.linear(input, self.weight, self.bias)

[mobilejammer]

the model I download from https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0

[mobilejammer]

https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0 can be train success!

[bghira]

normal sdxl works for me, your cache must have refiner objects in it.

[mobilejammer]

--override_dataset_config
Yes! I had configured a parameter --override_dataset_config before, but when I reran configure.py later, I didn't add it, which resulted in the cache field not being updated. Thank you for your help!