[Bug]: Cannot allocate 4GB block on ARC A770

[Krebzonide]

Checklist

• The issue has not been resolved by following the troubleshooting guide
• The issue exists on a clean installation of Fooocus
• The issue exists in the current version of Fooocus
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

When attempting to use a few different Fooocus tools, like 2x upscale or a SD 1.5 refiner, I get the error "RuntimeError: Current platform can NOT allocate memory block with size larger than 4GB!". I found someone else with the same issue using a different web ui here, and they seem to have fixed it, but I don't have the knowledge to implement this. lllyasviel/stable-diffusion-webui-forge#523

Steps to reproduce the problem

1. Select "Input Image"
2. Add an image
3. Select "Upscale (2x)"
4. Click "Generate"

What should have happened?

It should complete the task rather than giving an error.

What browsers do you use to access Fooocus?

Google Chrome

Where are you running Fooocus?

Locally

What operating system are you using?

Windows 10

Console logs

Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.09 seconds
  0%|                                                                                                                                                                                                              | 0/15 [00:00<?, ?it/s]2024-03-14 19:58:48,931 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 13%|██████████████████████████▍                                                                                                                                                                           | 2/15 [00:05<00:34,  2.67s/it]2024-03-14 19:58:50,239 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 20%|███████████████████████████████████████▌                                                                                                                                                              | 3/15 [00:07<00:24,  2.03s/it]2024-03-14 19:58:51,504 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 27%|████████████████████████████████████████████████████▊                                                                                                                                                 | 4/15 [00:08<00:18,  1.71s/it]2024-03-14 19:58:52,740 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 33%|██████████████████████████████████████████████████████████████████                                                                                                                                    | 5/15 [00:09<00:15,  1.54s/it]2024-03-14 19:58:53,972 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 40%|███████████████████████████████████████████████████████████████████████████████▏                                                                                                                      | 6/15 [00:10<00:12,  1.43s/it]2024-03-14 19:58:55,213 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 47%|████████████████████████████████████████████████████████████████████████████████████████████▍                                                                                                         | 7/15 [00:12<00:10,  1.37s/it]2024-03-14 19:58:56,420 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 53%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                            | 8/15 [00:13<00:09,  1.32s/it]2024-03-14 19:58:57,646 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 60%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                                                               | 9/15 [00:14<00:07,  1.29s/it]2024-03-14 19:58:58,892 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 67%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                                                 | 10/15 [00:15<00:06,  1.27s/it]2024-03-14 19:59:00,110 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 73%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                    | 11/15 [00:17<00:05,  1.26s/it]2024-03-14 19:59:01,333 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                       | 12/15 [00:18<00:03,  1.25s/it]2024-03-14 19:59:02,564 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 87%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                          | 13/15 [00:19<00:02,  1.24s/it]2024-03-14 19:59:03,787 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
 93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊             | 14/15 [00:20<00:01,  1.24s/it]2024-03-14 19:59:05,018 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:21<00:00,  1.46s/it]
Fooocus VAE-based swap.
2024-03-14 19:59:06,267 - httpx - INFO - HTTP Request: POST http://127.0.0.1:7865/api/predict "HTTP/1.1 200 OK"
Requested to load Interposer
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.30 seconds
Requested to load BaseModel
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.38 seconds
  0%|                                                                                                                                                                                                              | 0/15 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "C:\Users\corey\Fooocus\modules\async_worker.py", line 900, in worker
    handler(task)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\modules\async_worker.py", line 803, in handler
    imgs = pipeline.process_diffusion(
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\modules\default_pipeline.py", line 472, in process_diffusion
    sampled_latent = core.ksampler(
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\modules\core.py", line 308, in ksampler
    samples = ldm_patched.modules.sample.sample(model,
  File "C:\Users\corey\Fooocus\ldm_patched\modules\sample.py", line 100, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "C:\Users\corey\Fooocus\ldm_patched\modules\samplers.py", line 712, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\modules\sample_hijack.py", line 157, in sample_hacked
    samples = sampler.sample(model_wrap, sigmas, extra_args, callback_wrap, noise, latent_image, denoise_mask, disable_pbar)
  File "C:\Users\corey\Fooocus\ldm_patched\modules\samplers.py", line 557, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\k_diffusion\sampling.py", line 701, in sample_dpmpp_2m_sde_gpu
    return sample_dpmpp_2m_sde(model, x, sigmas, extra_args=extra_args, callback=callback, disable=disable, eta=eta, s_noise=s_noise, noise_sampler=noise_sampler, solver_type=solver_type)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\k_diffusion\sampling.py", line 613, in sample_dpmpp_2m_sde
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\corey\Fooocus\modules\patch.py", line 321, in patched_KSamplerX0Inpaint_forward
    out = self.inner_model(x, sigma,
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\modules\samplers.py", line 271, in forward
    return self.apply_model(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\modules\samplers.py", line 268, in apply_model
    out = sampling_function(self.inner_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
  File "C:\Users\corey\Fooocus\modules\patch.py", line 237, in patched_sampling_function
    positive_x0, negative_x0 = calc_cond_uncond_batch(model, cond, uncond, x, timestep, model_options)
  File "C:\Users\corey\Fooocus\ldm_patched\modules\samplers.py", line 222, in calc_cond_uncond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
  File "C:\Users\corey\Fooocus\ldm_patched\modules\model_base.py", line 85, in apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\corey\Fooocus\modules\patch.py", line 404, in patched_unet_forward
    h = forward_timestep_embed(module, h, emb, context, transformer_options, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\diffusionmodules\openaimodel.py", line 43, in forward_timestep_embed
    x = layer(x, context, transformer_options)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\attention.py", line 613, in forward
    x = block(x, context=context[i], transformer_options=transformer_options)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\attention.py", line 440, in forward
    return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\diffusionmodules\util.py", line 189, in checkpoint
    return func(*inputs)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\attention.py", line 500, in _forward
    n = self.attn1(n, context=context_attn1, value=value_attn1)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\corey\Fooocus\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\attention.py", line 392, in forward
    out = optimized_attention(q, k, v, self.heads)
  File "C:\Users\corey\Fooocus\ldm_patched\ldm\modules\attention.py", line 325, in attention_pytorch
    out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
RuntimeError: Current platform can NOT allocate memory block with size larger than 4GB! Tried to allocate 7.75 GiB (GPU  0; 15.56 GiB total capacity; 1.79 GiB already allocated; 1.86 GiB reserved in total by PyTorch)

Additional information

My GPU is an Arc A770

[mashb1t]

@Krebzonide Intel ARC GPUs are not officially supported (yet). Please connect with #2120, maybe there is a step missing in the setup.

[Krebzonide]

I asked Intel about the issue and apparently they already have a fork that fixes this issue, and speeds up the generation by about 30%. https://github.com/qiacheng/Fooocus

To help out future confused Arc users, here are the download commands specifically for Arc

git clone https://github.com/qiacheng/Fooocus
cd fooocus
python -m venv venv
.\venv\scripts\activate
pip install -r requirements_versions.txt
python.exe -m pip uninstall torch torchvision torchaudio torchtext functorch xformers -y
python.exe -m pip install "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/torch-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl"  "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/torchaudio-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl"  "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/torchvision-0.16.0a0+cxx11.abi-cp310-cp310-win_amd64.whl"  "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/intel_extension_for_pytorch-2.1.10+xpu-cp310-cp310-win_amd64.whl"
python entry_with_update.py

That last command will start it for the first time. For future starts throw this command into a run.bat file in the Fooocus folder, then you can shortcut it to put the launcher anywhere.

.\venv\Scripts\python.exe -s .\launch.py

[redrum-llik]

I asked Intel about the issue and apparently they already have a fork that fixes this issue, and speeds up the generation by about 30%. https://github.com/qiacheng/Fooocus

To help out future confused Arc users, here are the download commands specifically for Arc

git clone https://github.com/qiacheng/Fooocus
cd fooocus
python -m venv venv
.\venv\scripts\activate
pip install -r requirements_versions.txt
python.exe -m pip uninstall torch torchvision torchaudio torchtext functorch xformers -y
python.exe -m pip install "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/torch-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl"  "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/torchaudio-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl"  "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/torchvision-0.16.0a0+cxx11.abi-cp310-cp310-win_amd64.whl"  "https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10+xpu/intel_extension_for_pytorch-2.1.10+xpu-cp310-cp310-win_amd64.whl"
python entry_with_update.py

That last command will start it for the first time. For future starts throw this command into a run.bat file in the Fooocus folder, then you can shortcut it to put the launcher anywhere.

.\venv\Scripts\python.exe -s .\launch.py

That is rather curious - from what I could see, the issue about > 4 GB blocks allocation is still open on their repo. Can you please share any details on what exactly have they changed in this release, if you have those? In the meantime, I am going to test this out with my Forge installation to see if this has an effect.

[HubKing]

I asked Intel about the issue and apparently they already have a fork that fixes this issue, and speeds up the generation by about 30%. https://github.com/qiacheng/Fooocus

To help out future confused Arc users, here are the download commands specifically for Arc

Hmm. I was just trying basics of PyTorch, and I got that 4GB error when I tried to create a large matrix. Of course, my GPU has 16GB VRAM, so it was weird. I think I am using their latest versions, "torch 2.3.1+cxx11.abi". Are Intel going to fix this in the future version, so I don't have to do anything but waiting, or do I have to manually set up something?

[HubKing]

Seems like the issue is open at intel/intel-extension-for-pytorch#325