ControlNet scribble mode via API: cannot get it to work. Mask image to blame?

[trzy]

Hi,

I'm trying to use scribble mode to take a scribble + prompt and get an image but I just simply cannot get it to work. I use the normal SD checkpoint (the one that would be used for txt2img) and then use the /controlnet/txt2img API endpoint as below. I think the issue is the mask image. If I load it with (the base64 data of) a all-white or all-black image, it's equivalent to leaving that parameter off altogether. If I just use the input image (the scribble), it seems to do something almost correct, but I can't really be sure what's going on.

Any help would be greatly appreciated!

   const payload = {
        "enable_hr": false,
        "hr_scale" : 2,
        "hr_upscaler" : "Latent",
        "hr_second_pass_steps" : 0,
        "hr_resize_x": 0,
        "hr_resize_y": 0,
        "denoising_strength": 0.0,
        "firstphase_width": 0,
        "firstphase_height": 0,
        "prompt": "",
        "styles": [],
        "seed": -1,
        "subseed": -1,
        "subseed_strength": 0.0,
        "seed_resize_from_h": -1,
        "seed_resize_from_w": -1,
        "batch_size": 1,
        "n_iter": 1,
        "steps": 20,
        "cfg_scale": 7.0,
        "width": 512,
        "height": 512,
        "restore_faces": false,
        "tiling": false,
        "negative_prompt": "",
        "eta": 0,
        "s_churn": 0,
        "s_tmax": 0,
        "s_tmin": 0,
        "s_noise": 1,
        "override_settings": {},
        "override_settings_restore_afterwards": true,
        "sampler_name": "Euler a",
        "sampler_index": "Euler a",
        "script_name": null,
        "script_args": []
    };

    // Our params
    payload["prompt"] = prompt;
    payload["seed"] = 42;
    payload["cfg_scale"] = 9;   // 7?
    payload["steps"] = 40;
    payload["batch_size"] = 4;
    payload["controlnet_units"] = [
        {
            "input_image": inputImageBase64,
            "mask": _whiteMaskImage, // this is equivalent to omitting "mask" altogether
            //"mask": "",
            "module": "scribble",
            "model": "control_sd15_scribble [fef5e48e]",
            "weight": 1,
            "resize_mode": "Scale to Fit (Inner Fit)",
            "lowvram": false,
            "processor_res": 64,
            "threshold_a": 64,
            "threshold_b": 64,
            "guidance": 1,
            "guidance_start": 0,
            "guidance_end": 1,
            "guessmode": true
        }
    ];

    // Post request
    const urlParams = {
        host: _imageServerParams.host,
        port: _imageServerParams.port,
        path: "/controlnet/txt2img",
        method: "POST",
        headers: { "Content-Type": "application/json" }
    };

Thank you,

-- B.

[trzy]

Note: I realize these are deprecated but I'm apparently still on a commit from March during which they worked. I will update and migrate but I think the problem is not in the route.

[AllenEdgarPoe]

I send my json message like this:

        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "seed": -1,
        "subseed": -1,
        "subseed_strength": 0,
        "batch_size": 1,
        "n_iter": 1,
        "steps": 20,
        "cfg_scale": 7,
        "width": 768,
        "height": 768,
        "restore_faces": False,
        "eta": 0,
        "sampler_index": "Euler a",
        "controlnet_units": [{
            "enabled": True,
            "input_image": controlnet_img_string,
            "guessmode": False,
            "weight": 0.7,
            "module": 'canny',
            "model": 'control_sd21_canny',
            "pixel_perfect": True,
        }],
    }

And send the request like this.

    js = response.json()
    image = Image.open(io.BytesIO(base64.b64decode(js["images"][0])))

It works fine so try to replace the module and model as your preference.