Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get Error while optimizing SDXL of DirectML example #1165

Open
klin2024 opened this issue May 19, 2024 · 1 comment
Open

Get Error while optimizing SDXL of DirectML example #1165

klin2024 opened this issue May 19, 2024 · 1 comment
Labels
DirectML DirectML

Comments

@klin2024
Copy link

klin2024 commented May 19, 2024

Describe the bug
Get Error while optimizing SDXL

To Reproduce
Command: python stable_diffusion_xl.py --optimize

Expected behavior
Optimize successfully.

Olive logs

Optimizing unet
[2024-05-19 05:11:53,655] [INFO] [run.py:279:run] Loading Olive module configuration from: C:\Users\AMD\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\olive_config.json
[2024-05-19 05:11:53,665] [INFO] [accelerator_creator.py:224:create_accelerators] Running workflow on accelerator specs: gpu-dml
[2024-05-19 05:11:53,666] [INFO] [engine.py:107:initialize] Using cache directory: cache
[2024-05-19 05:11:53,666] [INFO] [engine.py:263:run] Running Olive on accelerator: gpu-dml
[2024-05-19 05:11:53,666] [INFO] [engine.py:1075:_create_system] Creating target system ...
[2024-05-19 05:11:53,667] [INFO] [engine.py:1078:_create_system] Target system created in 0.000997 seconds
[2024-05-19 05:11:53,667] [INFO] [engine.py:1087:_create_system] Creating host system ...
[2024-05-19 05:11:53,667] [INFO] [engine.py:1090:_create_system] Host system created in 0.000000 seconds
[2024-05-19 05:11:53,692] [INFO] [engine.py:865:_run_pass] Running pass convert:OnnxConversion
[2024-05-19 05:18:58,157] [INFO] [engine.py:952:_run_pass] Pass convert:OnnxConversion finished in 424.462611 seconds
[2024-05-19 05:18:58,168] [INFO] [engine.py:865:_run_pass] Running pass optimize:OrtTransformersOptimization
fusion: 100%|█████████████████████████████████████████████████████████████████████████| 18/18 [07:59<00:00, 26.61s/it]
Traceback (most recent call last):
File "D:\olive_0516\Olive\examples\directml\stable_diffusion_xl\stable_diffusion_xl.py", line 637, in
main()
File "D:\olive_0516\Olive\examples\directml\stable_diffusion_xl\stable_diffusion_xl.py", line 603, in main
optimize(
File "D:\olive_0516\Olive\examples\directml\stable_diffusion_xl\stable_diffusion_xl.py", line 369, in optimize
olive_run(olive_config)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\workflows\run\run.py", line 301, in run
return run_engine(package_config, run_config, data_root)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\workflows\run\run.py", line 255, in run_engine
engine.run(
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\engine\engine.py", line 265, in run
run_result = self.run_accelerator(
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\engine\engine.py", line 337, in run_accelerator
output_footprint = self.run_no_search(
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\engine\engine.py", line 429, in run_no_search
should_prune, signal, model_ids = self._run_passes(
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\engine\engine.py", line 827, in _run_passes
model_config, model_id = self._run_pass(
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\engine\engine.py", line 935, in _run_pass
output_model_config = host.run_pass(p, input_model_config, data_root, output_model_path, pass_search_point)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\systems\local.py", line 32, in run_pass
output_model = the_pass.run(model, data_root, output_model_path, point)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\passes\olive_pass.py", line 223, in run
output_model = self._run_for_config(model, data_root, config, output_model_path)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\passes\onnx\transformer_optimization.py", line 332, in run_for_config
return model_proto_to_olive_model(optimizer.model, output_model_path, config)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\passes\onnx\common.py", line 170, in model_proto_to_olive_model
has_external_data = model_proto_to_file(
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\olive\passes\onnx\common.py", line 114, in model_proto_to_file
onnx.save_model(model, str(output_path))
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\onnx_init
.py", line 327, in save_model
serialized = _get_serializer(format, model_filepath).serialize_proto(proto)
File "C:\Users\KT\miniconda3\envs\sdxl_turbo\lib\site-packages\onnx\serialization.py", line 100, in serialize_proto
result = proto.SerializeToString()
ValueError: Message onnx.ModelProto exceeds maximum protobuf size of 2GB: 5136056262

In model_proto_to_file, model_size was 841088966.

@klin2024 klin2024 changed the title Get Error while optimizing SDXL Get Error while optimizing SDXL of DirectML example May 19, 2024
@devang-ml devang-ml added the DirectML DirectML label Jun 3, 2024
@Jay19751103
Copy link

Hi Olive team
It looks that file size over 2GB during optimization pass. convert pass have flag to prevent this.
But the flag is not set in Transformation pass. I tried to add it can pass transformation under tag v0.6.2

Could any one fix this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DirectML DirectML
Projects
None yet
Development

No branches or pull requests

3 participants