This is an application that you can run in an IDE to create custom comics using Stable Diffusion and pretrained model provaided by ogkalu.
- text2img Stable Diffusion pipeline for generating images based on positive and negative prompt.
- img2img Stable Diffusion pipeline for generating images based on your simple painting as well as additional prompt to describe it.
- inpainting Stable Diffusion pipeline for mostly adding and fixing elements of generated image (paint and prompt part of image to fix).
- costum comic artstyle trained model provided by the user ogkalu on Huggingface.
Since everything works locally you have to firstly download inpainting and pretrained comic model for stable diffusion.
In main file you then have to set path at line 36:
pipelinePaint = StableDiffusionInpaintPipeline.from_pretrained(
r"inpainintg_parent_folder", revision="fp16",
torch_dtype=torch.float16)
and in modelID file set the comic path (example):
modelID = r"C:\Users\User\.cache\huggingface\diffusers\models--ogkalu--Comic-Diffusion\snapshots\b0f8b7655c342796dd3e25e2182f200919619e7c"
Finally copy the unet folder from Comic to inpainting (replace).
Note that at least 8GB of VRAM is needed (supports only Nividia Cuda GPU) as well as torch compiled with cuda!
Most things should be self explanatory. Know limitations are from pretrained model as well as consisterncy of charachters. I recommend prompting with known people like Jack Sparrow or John Wick to get some consistency. Comic is of size 3x4 (12 pictures) and can be exported as .pdf and printed in A4 format.
Look for inspiration on OpenArt for better prompting.
Example use of simple drawing.
As you can see from the image I have replaced the blue soldier's head as well as added blood to the body below.
Consistency of characters is achieved through proper prompting (i.e. by utilizing famous people and regenerating images with different prompts). If you want custom characters then additional training is needed which is currently not supported.
I am well aware that the application is not editor friendly for any modifications as it is all in one file.