How do we train lora for Flux-Fill-Dev?

[matabear-wyx]

The newly released Flux-Fill-Dev (Inpainting/Outpainting) is incredible. However, the LoRA I previously trained on Flux-Dev doesn't work directly with Flux-Fill-Dev, so I'm planning to retrain a LoRA specifically for Flux-Fill-Dev.

From my quick analysis, Flux-Fill-Dev takes an input of a 16-channel image latent, a 16-channel masked image latent, and a 64-channel mask. My initial idea for input preparation is to use the 16-channel image latent repeated twice, along with a 64-channel mask filled entirely with zeros (black mask).

Any thoughts or suggestions?

[chengyou-jia]

any new update for your training? i am also trying to train flux-fill-dev with lora.

[matabear-wyx]

Yes, you can try my method, it works properly

[chengyou-jia]

Thank you for your reply! How is the performance of the fill-model trained with LoRA? Would it be possible for you to share the code for the update? I would greatly appreciate it! 😊

[matabear-wyx]

What I did is really simple and raw, as I discussed:
`def pack_fill_latents(latents, batch_size, num_channels_latents, height, width):
latents = latents.view(
batch_size, num_channels_latents, height // 2, 2, width // 2, 2
)
latents = latents.permute(0, 2, 4, 1, 3, 5)
latents = latents.reshape(
batch_size, (height // 2) * (width // 2), num_channels_latents * 4
)
# Step 2: Repeat the packed latents
repeated_latents = latents.repeat(1, 1, 2) # Repeat along the channel dimension

# Step 3: Create a black mask of the required shape
mask = torch.zeros((batch_size, (height // 2) * (width // 2), 8 * 8 * 4), device=latents.device)

# Step 4: Concatenate repeated latents with the black mask
packed_fill_latents = torch.cat([repeated_latents, mask], dim=-1)

return packed_fill_latents`

[chengyou-jia]

thanks, so did you prepare a dataset(using real masks)and test it's effectiveness?

[Sebastian-Zok]

Hey, this sounds promising. Would you mind sharing your whole code? I can't seem to get the changes running with validation

[matabear-wyx]

Yes, my co-worker used a real mask for a repainting task, and it worked properly. However, I suggest following the Diffusers codebase to minimize coding effort. As you can see, what I did here was simply use a dummy mask.

[Bilal143260]

Yes, my co-worker used a real mask for a repainting task, and it worked properly. However, I suggest following the Diffusers codebase to minimize coding effort. As you can see, what I did here was simply use a dummy mask.

Which codebase are you exactly referring to? Is this the one for starting point ?
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_flux.py

[bc129697]

The newly released Flux-Fill-Dev (Inpainting/Outpainting) is incredible. However, the LoRA I previously trained on Flux-Dev doesn't work directly with Flux-Fill-Dev, so I'm planning to retrain a LoRA specifically for Flux-Fill-Dev.

From my quick analysis, Flux-Fill-Dev takes an input of a 16-channel image latent, a 16-channel masked image latent, and a 64-channel mask. My initial idea for input preparation is to use the 16-channel image latent repeated twice, along with a 64-channel mask filled entirely with zeros (black mask).

Any thoughts or suggestions?

你好，我最近也在用flux-fill-dev作为基座模型来训练，数据输入的channel也是16+16+64，但是训练出来的结果很差，想问一下你训练得怎么样了

[bc129697]

The newly released Flux-Fill-Dev (Inpainting/Outpainting) is incredible. However, the LoRA I previously trained on Flux-Dev doesn't work directly with Flux-Fill-Dev, so I'm planning to retrain a LoRA specifically for Flux-Fill-Dev.

From my quick analysis, Flux-Fill-Dev takes an input of a 16-channel image latent, a 16-channel masked image latent, and a 64-channel mask. My initial idea for input preparation is to use the 16-channel image latent repeated twice, along with a 64-channel mask filled entirely with zeros (black mask).

Any thoughts or suggestions?

Hello, I have recently been using flux-fill-dev as the base model for training. The data input channel is also 16+16+64, but the training results are very poor. I would like to ask about your training results.

[matabear-wyx]

Flux Fill is still a distilled model after all, which means it still can't learn new distributions. I'm not sure exactly what task you're working on, but generally speaking, I’d recommend training only specific layers of the LoRA instead of doing a full finetune.

[bghira]

distilled models can learn just fine.

[Sebastian-Zok]

Here's my training script. I used just one instance prompt describing the inpaining task, instead of a caption for each sample. The LoRa adapted my concept with just a couple of hundred steps. But there are some known issues: like the validation is broken and also the masking is hardcoded, but I hope it helps someone to adapt to their use cases :)