"We will stop supporting diff models soon because of its lack of robustness"

[Lexcess]

I saw that this message has started being emitted from a recent commit. Was there any discussion of this anywhere, or can there be here?

I have been using the diff models from when they became available. For me they have the following value:

• Halving the size of the model (from 1.4GiB to .7GiB)
  • This is great in local space saving
  • More importantly it directly relates to cost and deployment time when using services like RunPod.
• The value add of SafeTensors format being used.

I see lack of robustness is given as the driver, but I haven't had any issues with the diff models and will gladly accept any current level of imprecision I might be experiencing as an acceptable trade-off.

I know @lllyasviel made this change, so I am not sure if this conversation should be here or over in the main ControlNet repo, but given the message was added here I have gone with that.

[lllyasviel]

Not to confuse diff models with fp16 models. SafeTensors and 700MB models are all fp16 models.

We support official models (5.3G for 1.0, 1.4G for 1.0, 1.4G for 1.1) and fp16 models (0.7G for 1.0, 0.7G for 1.1) in pth, ckpt, or safetensors format. You can still use non-diff models with 700MB and SafeTensor format even if we remove support for diff models.

Diff model has nothing to do with model size and format - they are models with weight offsets obtained from your base model. The "diff" means the difference between controlnet and your base model. For example, if your base model is stable diffusion 1.5, then the diff means the difference between controlnet and stable diffusion 1.5. Then if your model is Realistic Vision, then a diff model will construct a controlnet by adding the diff to Realistic Vision.

Although many people do not know, but everytime you load a diff model, you are actually merging a new model from controlnet and your base model.

You may ask what is the difference and how it really works. The answer is that we do not know and this is the reason why we are going to end support of it because we cannot ensure its quality (but considering that this may conflict our principle to make sure that all previous results can be reproduced, perhaps we will never really remove it).

You can find non-diff version of 700MB controlnets 1.0 or 1.1 everywhere on internet and they wont emit any logs

[lllyasviel]

like this https://huggingface.co/comfyanonymous/ControlNet-v1-1_fp16_safetensors/tree/main

[lllyasviel]

Sigh – the diff model is just a mistake we made in the past and we need to explain something super complicated every time we run into issue. Perhaps this time I write this a bit longer and all future questions can use this copy and paste

[GeekyGhost]

So all my difference models aren't going to be usable if I update?

[lllyasviel]

So all my difference models aren't going to be usable if I update?

you can still use it but it will emit a log saying it is obsolate

[GeekyGhost]

Ahh okay lol, far less concerning of an issue lol. Thank you.

[lllyasviel]

We will begin to consider support diff models in a long term if any user succeeded in finding a reason why we should support it.
Note that you can find non-diff versions of 700MB safetensors models for CN 1.0 or 1.1 everywhere in internet. Those 700MB safetensor models are not diff models and wont emit any logs.

[GeekyGhost]

Ahh, it's okay. As long as they are still usable I can continue to support my own lol. If they got blocked from use somehow that would be different lol. Long as I can use them still I'm good. I have all the controlnet models, the diff ones just worrk the way I need lol. Lot of my animations and techniques rely on weird things only I do lol. This was one of them. Supporting them with the rest probably would only benefit me and I can handle that on my end.

[Lexcess]

Ok so is the reason for the default 1.4GiB models existing as they are is that they support older graphics cards? While the fp16 require newer (20XX?) cards? Does that mean that bf16 could be even smaller?

On the diff models specifically is the implication that a fp16 diff model would be halved again (~350mb). Because that would be a selling point to me, given how many models there are and how many you sometimes end up applying in one run. IO to container services is often the biggest delay unless you are doing super long runs (video etc perhaps).

Lastly, excuse my ignorance, but is there any reason not to deliver all the models as safetensors now? I thought most of the GUIs out there now support it (and those that do not probably shoild right?).

[lllyasviel]

no. your diff model is already fp16. it is not possible to reduce it to 300m.

[Lexcess]

And the reason for the default models not being fp16 in SafeTensor format is for backwards compatibility or something similar?

[lllyasviel]

"modules.devices.NansException: A tensor with all NaNs was produced in VAE. This could be because there's no enough precision to represent this picture. Try adding --no-half-vae commandline to fix this."

[Lexcess]

@lllyasviel I think you replied to the wrong thread

[Vendaciousness]

I got this message when I used the TemporalNet diff model. The thing about that model, which I find quite useful for my work, is that I don't have enough VRAM to use the regular model, which is 12GB. I noticed there was also something about 'stop using custom models' (I don't have the message in front of me atm). Are you going to be blocking the use of non standard models in the future? This is important, as all I use at the moment are non-standard models.

[nblive15]

I received too
THe diff temporalnet is very usefull
I hope they keep working