Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Issue]: Folder for Huggingface cache gets reset #3698

Closed
2 tasks done
IowaSovereign opened this issue Jan 12, 2025 · 6 comments
Closed
2 tasks done

[Issue]: Folder for Huggingface cache gets reset #3698

IowaSovereign opened this issue Jan 12, 2025 · 6 comments
Labels
question Further information is requested

Comments

@IowaSovereign
Copy link

Issue Description

After manually setting folder for Huggingface cache (hfcache_dir) it gets removed from the config returning to default /root/.cache/huggingface/hub after full server restart.

Version Platform Description

20:01:52-573783 INFO Starting SD.Next
20:01:52-587441 INFO Logger: file="/workspace/SD_Next/sdnext.log" level=DEBUG size=64 mode=create
20:01:52-958467 INFO Version: app=sd.next updated=2025-01-11 hash=389bb78b branch=dev url=https://github.com/vladmandic/automatic.git/tree/dev ui=dev
20:01:52-996920 INFO Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.5.0-35-generic python=3.10.12 docker=False
20:01:53-014444 DEBUG Packages: venv=venv site=['venv/lib/python3.10/site-packages']
20:01:53-020014 INFO Args: ['--listen', '--port', '3031', '--insecure', '--quick', '--debug']

Relevant log output

No response

Backend

Diffusers

UI

ModernUI

Branch

Dev

Model

StableDiffusion XL

Acknowledgements

  • I have read the above and searched for existing issues
  • I confirm that this is classified correctly and its not an extension issue
@vladmandic
Copy link
Owner

vladmandic commented Jan 12, 2025 •
edited
Loading

show that behavior - logs, anything? i may believe you, but nothing i can do based just on a general statement.
also, upload output of env command?

@vladmandic vladmandic added the question Further information is requested label Jan 12, 2025
@IowaSovereign
Copy link
Author

IowaSovereign commented Jan 12, 2025 •
edited by vladmandic
Loading

Oh, I'm so sorry, somehow the log didn't upload? Please find below. It's very long, so I'll have to split it in two.

This log shows the first startup, in which I set the path. I can confirm last line of config.json is:
hfcache_dir"/workspace/SD_Next/models/huggingface"

14:24:53-769821 INFO     Starting SD.Next                                                                                                                                                               
14:24:53-784868 INFO     Logger: file="/workspace/SD_Next/sdnext.log" level=DEBUG size=64 mode=create                                                                                                   
14:24:53-791948 INFO     Python: version=3.10.12 platform=Linux bin="/workspace/SD_Next/venv/bin/python3" venv="/workspace/SD_Next/venv"                                                                
14:24:54-049504 INFO     Version: app=sd.next updated=2025-01-11 hash=b4575ad8 branch=dev url=https://github.com/vladmandic/automatic.git/tree/dev ui=dev                                               
14:24:54-679834 INFO     Repository latest available 3f36089142d6f71b486ab39f316405946aee8752 2025-01-02T21:54:48Z                                                                                      
14:24:54-714097 INFO     Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.5.0-35-generic python=3.10.12 docker=False                                                                             
14:24:54-721962 DEBUG    Packages: venv=venv site=['venv/lib/python3.10/site-packages']                                                                                                                 
14:24:54-728951 INFO     Args: ['--listen', '--port', '3031', '--debug', '--safe']                                                                                                                      
14:24:54-733077 DEBUG    Setting environment tuning                                                                                                                                                     
14:24:54-737670 DEBUG    Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"                                                                                                     
14:24:54-768643 DEBUG    Torch overrides: cuda=False rocm=False ipex=False directml=False openvino=False zluda=False                                                                                    
14:24:54-775937 INFO     CUDA: nVidia toolkit detected                                                                                                                                                                                                                   
14:24:57-056056 INFO     Install: verifying requirements                                                                                                                                                
14:24:57-075484 INFO     Verifying packages                                                                                                                                                             
14:24:57-164633 DEBUG    Timestamp repository update time: Sat Jan 11 23:51:33 2025                                                                                                                     
14:24:57-168839 INFO     Startup: standard                                                                                                                                                              
14:24:57-172572 INFO     Verifying submodules                                                                                                                                                           
14:25:01-585971 DEBUG    Git submodule: extensions-builtin/sd-extension-chainner / main                                                                                                                 
14:25:01-829270 DEBUG    Git submodule: extensions-builtin/sd-extension-system-info / main                                                                                                              
14:25:01-980816 DEBUG    Git submodule: extensions-builtin/sd-webui-agent-scheduler / main                                                                                                              
14:25:02-167798 DEBUG    Git submodule: extensions-builtin/sdnext-modernui / dev                                                                                                                        
14:25:02-401456 DEBUG    Git submodule: extensions-builtin/stable-diffusion-webui-rembg / master                                                                                                        
14:25:02-565275 DEBUG    Git submodule: modules/k-diffusion / master                                                                                                                                    
14:25:02-733658 DEBUG    Git submodule: wiki / master                                                                                                                                                   
14:25:02-845625 DEBUG    Register paths                                                                                                                                                                 
14:25:03-625226 DEBUG    Installed packages: 282                                                                                                                                                        
14:25:03-631015 DEBUG    Extensions all: ['sdnext-modernui', 'stable-diffusion-webui-rembg', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-extension-chainner', 'Lora']                   
14:25:04-371698 DEBUG    Extension installer: /workspace/SD_Next/extensions-builtin/stable-diffusion-webui-rembg/install.py                                                                             
14:25:24-469770 DEBUG    Extension installer: /workspace/SD_Next/extensions-builtin/sd-webui-agent-scheduler/install.py                                                                                 
14:25:36-101107 INFO     Extensions enabled: ['sdnext-modernui', 'stable-diffusion-webui-rembg', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-extension-chainner', 'Lora']               
14:25:36-118172 INFO     Install: verifying requirements                                                                                                                                                
14:25:36-124931 DEBUG    Setup complete without errors: 1736691936                                                                                                                                      
14:25:36-129519 INFO     Running in safe mode without user extensions                                                                                                                                   
14:25:36-162191 DEBUG    Extension preload: {'extensions-builtin': 0.01}                                                                                                                                
14:25:36-170151 INFO     Command line args: ['--listen', '--port', '3031', '--debug', '--safe'] listen=True port=3031 safe=True debug=True args=[]                                                      
14:25:36-175927 DEBUG    Env flags: ['SD_CONFIG_DEBUG=true=true']                                                                                                                                       
14:25:36-180752 DEBUG    Starting module: <module 'webui' from '/workspace/SD_Next/webui.py'>                                                                                                           
14:26:03-559245 INFO     Device detect: memory=44.3 default=balanced                                                                                                                                    
14:26:03-579892 DEBUG    Read: file="config.json" json=42 bytes=1838 time=0.001 fn=<module>:load                                                                                                        
14:26:04-568292 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda device=cuda attention="Scaled-Dot-Product" mode=no_grad                                                                         
14:26:04-575687 DEBUG    Read: file="html/reference.json" json=62 bytes=32535 time=0.002 fn=_call_with_frames_removed:<module>                                                                          
14:26:04-662533 INFO     Torch parameters: backend=cuda device=cuda config=Auto dtype=torch.bfloat16 context=no_grad nohalf=False nohalfvae=False upcast=False deterministic=False fp16=pass bf16=pass  
                         optimization="Scaled-Dot-Product"                                                                                                                                              
14:26:05-443711 DEBUG    ONNX: version=1.19.0 provider=CUDAExecutionProvider, available=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']                                  
14:26:05-870691 DEBUG    ONNX failed to initialize XL pipelines: module 'optimum.onnxruntime.modeling_diffusion' has no attribute 'ORTPipelinePart'                                                     
14:26:05-945230 INFO     Device: device=NVIDIA A40 n=1 arch=sm_90 capability=(8, 6) cuda=12.1 cudnn=90100 driver=550.127.05                                                                             
14:26:13-527510 DEBUG    System: cores=96 affinity=96 threads=48                                                                                                                                        
14:26:13-537044 INFO     Torch: torch==2.4.0+cu121 torchvision==0.19.0+cu121                                                                                                                            
14:26:13-541780 INFO     Packages: diffusers==0.33.0.dev0 transformers==4.47.1 accelerate==1.2.1 gradio==3.43.2                                                                                         
14:26:13-886110 DEBUG    Entering start sequence                                                                                                                                                        
14:26:13-919184 DEBUG    Initializing                                                                                                                                                                   
14:26:13-932968 DEBUG    Read: file="metadata.json" json=246 bytes=593555 time=0.008 fn=initialize:init_metadata                                                                                        
14:26:13-941973 DEBUG    Read: file="cache.json" json=2 bytes=13730 time=0.002 fn=initialize:init_cache                                                                                                 
14:26:13-946955 DEBUG    Huggingface cache: path="/root/.cache/huggingface/hub"                                                                                                                         
14:26:14-146253 INFO     Available VAEs: path="models/VAE" items=11                                                                                                                                     
14:26:14-200412 INFO     Available UNets: path="models/UNET" items=37                                                                                                                                   
14:26:14-227394 INFO     Available TEs: path="models/Text-encoder" items=9                                                                                                                              
14:26:14-376085 INFO     Available Models: items=42 safetensors="models/Stable-diffusion":37 diffusers="models/diffusers":5 time=0.14                                                                   
14:26:14-496270 INFO     Available LoRAs: path="models/Lora" items=84 folders=8 time=0.06                                                                                                               
14:26:14-535970 INFO     Available Styles: folder="models/styles" items=20 time=0.04                                                                                                                    
14:26:15-029772 INFO     Available Yolo: path="models/yolo" items=7 downloaded=4                                                                                                                        
14:26:15-039552 DEBUG    Extensions: disabled=['Lora', 'sd-extension-chainner', 'sd-webui-agent-scheduler', 'stable-diffusion-webui-rembg']                                                             
14:26:15-043662 INFO     Load extensions                                                                                                                                                                
14:26:16-900824 DEBUG    Extensions init time: total=1.85 pulid_ext.py=1.41                                                                                                                             
14:26:16-958892 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2640 time=0.001 fn=__init__:__init__                                                                                             
14:26:16-970046 DEBUG    Upscaler type=ESRGAN folder="models/ESRGAN" model="DAT-Helaman-Nomos-4x" path="models/ESRGAN/DAT-Helaman-Nomos-4x.pth"                                                         
14:26:16-997955 INFO     Available Upscalers: items=29 downloaded=6 user=1 time=0.07 types=['None', 'Lanczos', 'Nearest', 'ESRGAN', 'SwinIR', 'AuraSR', 'SD', 'SCUNet', 'RealESRGAN']                   
14:26:17-032248 DEBUG    UI start sequence                                                                                                                                                              
14:26:17-036292 INFO     UI theme: type=Modern name="Default" available=34                                                                                                                              
14:26:17-076958 DEBUG    UI theme: css="extensions-builtin/sdnext-modernui/themes/Default.css" base="base.css" user="None"                                                                              
14:26:17-084501 DEBUG    UI initialize: txt2img                                                                                                                                                         
14:26:17-790545 DEBUG    Networks: page='model' items=103 subfolders=7 tab=txt2img folders=['models/Stable-diffusion', 'models/diffusers', 'models/Reference'] list=0.55 thumb=0.21 desc=0.22 info=0.30 
                         workers=8                                                                                                                                                                      
14:26:17-940826 DEBUG    Networks: page='lora' items=84 subfolders=8 tab=txt2img folders=['models/Lora'] list=0.54 thumb=0.25 desc=0.27 info=0.93 workers=8                                             
14:26:17-971982 DEBUG    Networks: page='style' items=20 subfolders=2 tab=txt2img folders=['models/styles', 'html'] list=0.27 thumb=0.25 desc=0.00 info=0.00 workers=8                                  
14:26:17-983509 DEBUG    Networks: page='embedding' items=0 subfolders=0 tab=txt2img folders=['models/embeddings'] list=0.01 thumb=0.01 desc=0.00 info=0.00 workers=8                                   
14:26:17-995511 DEBUG    Networks: page='vae' items=11 subfolders=0 tab=txt2img folders=['models/VAE'] list=0.36 thumb=0.11 desc=0.01 info=0.01 workers=8                                               
14:26:18-002743 DEBUG    Networks: page='history' items=0 subfolders=0 tab=txt2img folders=[] list=0.00 thumb=0.00 desc=0.00 info=0.00 workers=8                                                        
14:26:18-650684 DEBUG    UI initialize: img2img                                                                                                                                                         
14:26:18-879190 DEBUG    UI initialize: control models=models/control                                                                                                                                   
14:26:20-016048 DEBUG    Read: file="ui-config.json" json=27 bytes=1596 time=0.007 fn=__init__:read_from_file                                                                                           
14:26:21-346997 DEBUG    Extension list: processed=397 installed=6 enabled=2 disabled=4 visible=397 hidden=0                                                                                            
14:26:21-821614 DEBUG    Root paths: ['/workspace/SD_Next']                                                                                                                                             
14:26:22-033793 INFO     Local URL: http://localhost:3031/                                                                                                                                              
14:26:22-044700 DEBUG    API middleware: [<class 'starlette.middleware.base.BaseHTTPMiddleware'>, <class 'starlette.middleware.gzip.GZipMiddleware'>]                                                   
14:26:22-060215 DEBUG    API initialize                                                                                                                                                                 
14:26:22-268683 DEBUG    SD-System-Info: benchmark data loaded: /workspace/SD_Next/extensions-builtin/sd-extension-system-info/scripts/benchmark-data-local.json                                        
14:26:22-340620 DEBUG    Scripts setup: time=0.774 ['K-Diffusion Samplers:0.367', 'XYZ Grid:0.061', 'IP Adapters:0.054', 'Face: Multiple ID Transfers:0.032', 'FreeScale: Tuning-Free Scale             
                         Fusion:0.016', 'Video: CogVideoX:0.015', 'Video: LTX Video:0.014', 'Video: AnimateDiff:0.013', 'ConsiStory: Consistent Image Generation:0.013', 'Video: VGen                   
                         Image-to-Video:0.012', 'PuLID: ID Customization:0.011', 'Ctrl-X: Controlling Structure and Appearance:0.011', 'LUT Color grading:0.011', 'Style Aligned Image Generation:0.01']
14:26:22-344483 DEBUG    Model metadata: file="metadata.json" no changes                                                                                                                                
14:26:22-347879 DEBUG    Model requested: fn=run:<lambda>                                                                                                                                               
14:26:22-351091 INFO     Load model: select="diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789]"                                                                                                       
14:26:22-363254 DEBUG    Load model: type=FLUX model="diffusers/black-forest-labs/FLUX.1-dev" repo="black-forest-labs/FLUX.1-dev" unet="None" te="None" vae="Automatic" quant=none offload=balanced     
                         dtype=torch.bfloat16                                                                                                                                                           
14:26:22-471479 INFO     HF login: token="/root/.cache/huggingface/token"                                                                                                                               
14:26:22-725287 DEBUG    GC: current={'gpu': 0.56, 'ram': 1.02, 'oom': 0} prev={'gpu': 0.57, 'ram': 1.02} load={'gpu': 1, 'ram': 0} gc={'gpu': 0.01, 'py': 5159} fn=load_diffuser_force:load_flux       
                         why=force time=0.25                                                                                                                                                            
14:26:22-728730 DEBUG    Load model: type=FLUX cls=FluxPipeline preloaded=[] revision=None                                                                                                              
Diffusers  3.82it/s █████████████ 100% 2/2 00:00 00:00 Loading checkpoint shards
Diffusers  2.81it/s ████████ 100% 7/7 00:02 00:00 Loading pipeline components...
14:26:25-393985 DEBUG    Setting model: component=VAE slicing=True                                                                                                                                      
14:26:25-396753 DEBUG    Setting model: attention="Scaled-Dot-Product"                                                                                                                                  
14:26:25-417303 INFO     Offload: type=balanced op=init watermark=0.5-0.9 gpu=22.170-39.906000000000006:44.34 cpu=503.510 limit=0.00                                                                    
14:26:25-434874 DEBUG    Model module=transformer type=FluxTransformer2DModel dtype=torch.bfloat16 quant=None params=11.084 size=22.168                                                                 
14:26:25-439518 DEBUG    Model module=text_encoder_2 type=T5EncoderModel dtype=torch.bfloat16 quant=None params=4.435 size=8.870                                                                        
14:26:25-442908 DEBUG    Model module=text_encoder type=CLIPTextModel dtype=torch.bfloat16 quant=None params=0.115 size=0.229                                                                           
14:26:25-446030 DEBUG    Model module=vae type=AutoencoderKL dtype=torch.bfloat16 quant=None params=0.078 size=0.156                                                                                    
14:26:25-448851 INFO     Model class=FluxPipeline modules=4 size=31.424                                                                                                                                 
14:26:25-455681 INFO     Load model: time=total=3.10 load=3.04 native=1024 memory={'ram': {'used': 1.14, 'total': 503.51}, 'gpu': {'used': 0.56, 'total': 44.34}, 'retries': 0, 'oom': 0}               
14:26:25-459332 DEBUG    Settings: user                                                                                                                                                                 
14:26:25-460258 DEBUG    Script init: ['system-info.py:app_started=0.09']                                                                                                                               
14:26:25-462724 TRACE      Config: item=sdp_options value=['Flash attention', 'Memory attention', 'Math attention'] default=['Flash attention', 'Memory attention', 'Math attention']                   
14:26:25-464882 INFO     Startup time: total=93.61 torch=81.67 launch=44.62 installer=44.33 checkpoint=3.12 extensions=1.85 ui-networks=1.17 ui-extensions=0.94 ui-txt2img=0.60 ui-models=0.58          
                         detailer=0.50 ui-settings=0.47 libraries=0.36 ui-defaults=0.35 ui-control=0.32 samplers=0.24 api=0.20 ui-img2img=0.18 models=0.15 lora=0.12 upscalers=0.12 app-started=0.09    
                         ui-extras=0.07 unet=0.07 ui-gallery=0.05                                                                                                                                       
14:26:25-469387 TRACE      Config: item=xformers_options value=['Flash attention'] default=['Flash attention']                                                                                          
14:26:25-474693 TRACE      Config: item=ipex_optimize value=[] default=[]                                                                                                                               
14:26:25-477225 TRACE      Config: item=openvino_devices value=[] default=[]                                                                                                                            
14:26:25-479280 TRACE      Config: item=bnb_quantization value=[] default=[]                                                                                                                            
14:26:25-481923 TRACE      Config: item=optimum_quanto_weights value=[] default=[]                                                                                                                      
14:26:25-484088 TRACE      Config: item=torchao_quantization value=[] default=[]                                                                                                                        
14:26:25-486623 TRACE      Config: item=nncf_compress_weights value=[] default=[]                                                                                                                       
14:26:25-488683 TRACE      Config: item=nncf_quantize value=[] default=[]                                                                                                                               
14:26:25-490849 TRACE      Config: item=cuda_compile value=[] default=[]                                                                                                                                
14:26:25-492973 TRACE      Config: item=styles_dir value=models/styles default=styles.csv                                                                                                               
14:26:25-495182 TRACE      Config: item=quicksettings_list value=['sd_model_checkpoint', 'sd_model_refiner', 'sd_unet', 'sd_vae', 'sd_text_encoder', 'show_progress_type']                              
                         default=['sd_model_checkpoint']                                                                                                                                                
14:26:25-497963 TRACE      Config: item=show_samplers value=[] default=[]                                                                                                                               
14:26:25-500284 TRACE      Config: item=postprocessing_enable_in_main_ui value=[] default=[]                                                                                                            
14:26:25-514936 TRACE      Config: item=postprocessing_operation_order value=[] default=[]                                                                                                              
14:26:25-517616 TRACE      Config: item=detailer_models value=['face-yolo8n', 'eyes-full-v1', 'hand_yolov8n'] default=['face-yolo8n']                                                                   
14:26:25-520081 TRACE      Config: item=interrogate_clip_skip_categories value=['artists', 'movements', 'flavors'] default=['artists', 'movements', 'flavors']                                          
14:26:25-522551 TRACE      Config: item=extra_networks value=['All'] default=['All']                                                                                                                    
14:26:25-524977 TRACE      Config: item=diffusers_version value=03bcf5aefef13a064c34b605e489c0730052cca8 default=                                                                                       
14:26:25-527391 TRACE      Config: item=disabled_extensions value=[] default=[]                                                                                                                         
14:26:25-529739 TRACE      Config: item=detailer_conf value=0.5 default=0.6                                                                                                                             
14:26:25-532326 TRACE      Config: item=detailer_max value=4 default=2                                                                                                                                  
14:26:25-534832 TRACE      Config: item=theme_type value=Modern default=Standard                                                                                                                        
14:26:25-537464 TRACE      Config: item=keep_incomplete value=False default=True                                                                                                                        
14:26:25-540011 TRACE      Config: item=samples_format value=png default=jpg                                                                                                                            
14:26:25-542469 TRACE      Config: item=save_log_fn value=/workspace/SD_Next/outputs/jobs.json default=                                                                                                 
14:26:25-545213 TRACE      Config: item=directories_filename_pattern value=[model_name] default=[date]                                                                                                  
14:26:25-548080 TRACE      Config: item=samples_filename_pattern value=[datetime<%Y-%m-%d_%H-%M-%S_%f>]-[model_name]-[sampler] default=[seq]-[model_name]-[prompt_words]                                
14:26:25-551933 TRACE      Config: item=live_preview_taesd_layers value=2 default=3                                                                                                                     
14:26:25-554611 TRACE      Config: item=interrogate_return_ranks value=False default=True                                                                                                               
14:26:25-557872 TRACE      Config: item=huggingface_token value=xxxxx default=                                                                                          
14:26:25-560634 TRACE      Config: item=lora_in_memory_limit value=3 default=0                                                                                                                          
14:26:25-563588 TRACE      Config: item=extra_networks_styles value=False default=True                                                                                                                  
14:26:25-567576 TRACE      Config: item=gradio_theme value=Default default=black-teal                                                                                                                   
14:26:25-570330 TRACE      Config: item=sd_model_checkpoint value=diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789] default=model.safetensors                                                         
14:26:25-573381 TRACE      Config: item=sd_checkpoint_hash value=None default=                                                                                                                          
14:26:25-576026 TRACE      Config: item=lora_fuse_diffusers value=False default=True                                                                                                                    
14:26:25-578812 TRACE      Config: item=diffusers_dir value=models/diffusers default=models/Diffusers                                                                                                   
14:26:25-582089 TRACE      Config: item=diffusers_offload_min_gpu_memory value=0.5 default=0.25                                                                                                         
14:26:25-585063 TRACE      Config: item=diffusers_offload_max_gpu_memory value=0.9 default=0.7                                                                                                          
14:26:25-587938 TRACE      Config: item=show_progress_type value=TAESD default=Approximate                                                                                                              
14:26:25-591038 TRACE      Config: item=save_to_dirs value=True default=False                                                                                                                           
14:26:25-593967 TRACE      Config: item=clip_skip value=1 default=None                                                                                                                                  
14:26:25-597069 TRACE      Config: item=uni_pc_lower_order_final value=True default=None                                                                                                                
14:26:25-600496 TRACE      Config: item=uni_pc_order value=2 default=None                                                                                                                               
14:26:25-615027 TRACE      Config: item=ckpt_dir value=models/Stable-diffusion default=models/Stable-diffusion                                                                                          
14:26:25-618193 TRACE      Config: item=vae_dir value=models/VAE default=models/VAE                                                                                                                     
14:26:25-621720 TRACE      Config: item=unet_dir value=models/UNET default=models/UNET                                                                                                                  
14:26:25-624644 TRACE      Config: item=te_dir value=models/Text-encoder default=models/Text-encoder                                                                                                    
14:26:25-627870 TRACE      Config: item=lora_dir value=models/Lora default=models/Lora                                                                                                                  
14:26:25-632096 TRACE      Config: item=embeddings_dir value=models/embeddings default=models/embeddings                                                                                                
14:26:25-635243 TRACE      Config: item=hypernetwork_dir value=models/hypernetworks default=models/hypernetworks                                                                                        
14:26:25-638514 TRACE      Config: item=onnx_temp_dir value=models/ONNX/temp default=models/ONNX/temp                                                                                                   
14:26:25-642339 TRACE      Config: item=outdir_txt2img_samples value=outputs/text default=outputs/text                                                                                                  
14:26:25-645533 TRACE      Config: item=outdir_img2img_samples value=outputs/image default=outputs/image                                                                                                
14:26:25-649009 TRACE      Config: item=outdir_control_samples value=outputs/control default=outputs/control                                                                                            
14:26:25-652311 TRACE      Config: item=outdir_extras_samples value=outputs/extras default=outputs/extras                                                                                               
14:26:25-658259 TRACE      Config: item=outdir_init_images value=outputs/init-images default=outputs/init-images                                                                                        
14:26:25-661987 TRACE      Config: item=outdir_txt2img_grids value=outputs/grids default=outputs/grids                                                                                                  
14:26:25-665180 TRACE      Config: item=outdir_img2img_grids value=outputs/grids default=outputs/grids                                                                                                  
14:26:25-668530 TRACE      Config: item=outdir_control_grids value=outputs/grids default=outputs/grids                                                                                                  
14:26:25-672127 TRACE      Config: item=outdir_save value=outputs/save default=outputs/save                                                                                                             
14:26:25-675400 TRACE      Config: item=outdir_video value=outputs/video default=outputs/video                                                                                                          
14:26:25-678843 TRACE      Config: item=yolo_dir value=models/yolo default=models/yolo                                                                                                                  
14:26:25-682458 TRACE      Config: item=wildcards_dir value=models/wildcards default=models/wildcards                                                                                                   
14:26:25-691637 DEBUG    Save: file="config.json" json=42 bytes=1838 time=0.006                                                                                                                         
14:26:25-696722 TRACE    Settings save: {'sdp_options': ['Flash attention', 'Memory attention', 'Math attention'], 'xformers_options': ['Flash attention'], 'ipex_optimize': [], 'openvino_devices': [],
                         'bnb_quantization': [], 'optimum_quanto_weights': [], 'torchao_quantization': [], 'nncf_compress_weights': [], 'nncf_quantize': [], 'cuda_compile': [], 'styles_dir':          
                         'models/styles', 'quicksettings_list': ['sd_model_checkpoint', 'sd_model_refiner', 'sd_unet', 'sd_vae', 'sd_text_encoder', 'show_progress_type'], 'show_samplers': [],         
                         'postprocessing_enable_in_main_ui': [], 'postprocessing_operation_order': [], 'detailer_models': ['face-yolo8n', 'eyes-full-v1', 'hand_yolov8n'],                              
                         'interrogate_clip_skip_categories': ['artists', 'movements', 'flavors'], 'extra_networks': ['All'], 'diffusers_version': '03bcf5aefef13a064c34b605e489c0730052cca8',           
                         'disabled_extensions': [], 'detailer_conf': 0.5, 'detailer_max': 4, 'theme_type': 'Modern', 'keep_incomplete': False, 'samples_format': 'png', 'save_log_fn':                  
                         '/workspace/SD_Next/outputs/jobs.json', 'directories_filename_pattern': '[model_name]', 'samples_filename_pattern': '[datetime<%Y-%m-%d_%H-%M-%S_%f>]-[model_name]-[sampler]', 
                         'live_preview_taesd_layers': 2, 'interrogate_return_ranks': False, 'huggingface_token': 'xxxx', 'lora_in_memory_limit': 3,                    
                         'extra_networks_styles': False, 'gradio_theme': 'Default', 'sd_model_checkpoint': 'diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789]', 'sd_checkpoint_hash': None,           
                         'lora_fuse_diffusers': False, 'diffusers_dir': 'models/diffusers', 'diffusers_offload_min_gpu_memory': 0.5, 'diffusers_offload_max_gpu_memory': 0.9, 'show_progress_type':     
                         'TAESD', 'save_to_dirs': True}                                                                                                                                                 
14:26:35-694838 INFO     API None 200 http/1.1 GET /sdapi/v1/motd 141.98.252.234 0.4941                                                                                                                 
14:26:45-265297 INFO     Browser session: user=None client=141.98.252.234 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36         
                         Edg/131.0.0.0                                                                                                                                                                  
14:26:45-270328 INFO     API None 200 http/1.1 GET /sdapi/v1/start 141.98.252.234 0.0062                                                                                                                
14:26:45-300734 INFO     API None 200 http/1.1 GET /sdapi/v1/sd-models 141.98.252.234 0.0458                                                                                                            
14:26:55-254920 INFO     UI: ready time=21.279                                                                                                                                                          
14:27:56-971338 TRACE    Settings set: hfcache_dir=/workspace/SD_Next/models/huggingface                                                                                                                
14:27:56-980683 DEBUG    Settings: user                                                                                                                                                                 
14:27:56-983011 INFO     Settings: changed=1 ['hfcache_dir']                                                                                                                                            
14:27:56-987277 TRACE      Config: item=sdp_options value=['Flash attention', 'Memory attention', 'Math attention'] default=['Flash attention', 'Memory attention', 'Math attention']                   
14:27:56-997165 TRACE      Config: item=xformers_options value=['Flash attention'] default=['Flash attention']                                                                                          
14:27:57-000063 TRACE      Config: item=ipex_optimize value=[] default=[]                                                                                                                               
14:27:57-014028 TRACE      Config: item=openvino_devices value=[] default=[]                                                                                                                            
14:27:57-017663 TRACE      Config: item=bnb_quantization value=[] default=[]                                                                                                                            
14:27:57-020689 TRACE      Config: item=optimum_quanto_weights value=[] default=[]                                                                                                                      
14:27:57-023604 TRACE      Config: item=torchao_quantization value=[] default=[]                                                                                                                        
14:27:57-027075 TRACE      Config: item=nncf_compress_weights value=[] default=[]                                                                                                                       
14:27:57-053132 TRACE      Config: item=nncf_quantize value=[] default=[]                                                                                                                               
14:27:57-056682 TRACE      Config: item=cuda_compile value=[] default=[]                                                                                                                                
14:27:57-060009 TRACE      Config: item=styles_dir value=models/styles default=styles.csv                                                                                                               
14:27:57-062961 TRACE      Config: item=quicksettings_list value=['sd_model_checkpoint', 'sd_model_refiner', 'sd_unet', 'sd_vae', 'sd_text_encoder', 'show_progress_type']                              
                         default=['sd_model_checkpoint']                                                                                                                                                
14:27:57-067269 TRACE      Config: item=show_samplers value=[] default=[]                                                                                                                               
14:27:57-070696 TRACE      Config: item=postprocessing_enable_in_main_ui value=[] default=[]                                                                                                            
14:27:57-074163 TRACE      Config: item=postprocessing_operation_order value=[] default=[]                                                                                                              
14:27:57-078264 TRACE      Config: item=detailer_models value=['face-yolo8n', 'eyes-full-v1', 'hand_yolov8n'] default=['face-yolo8n']                                                                   
14:27:57-082999 TRACE      Config: item=interrogate_clip_skip_categories value=['artists', 'movements', 'flavors'] default=['artists', 'movements', 'flavors']                                          
14:27:57-088231 TRACE      Config: item=extra_networks value=['All'] default=['All']                                                                                                                    
14:27:57-092719 TRACE      Config: item=diffusers_version value=03bcf5aefef13a064c34b605e489c0730052cca8 default=                                                                                       
14:27:57-097194 TRACE      Config: item=disabled_extensions value=[] default=[]                                                                                                                         
14:27:57-112378 TRACE      Config: item=detailer_conf value=0.5 default=0.6                                                                                                                             
14:27:57-118063 TRACE      Config: item=detailer_max value=4 default=2                                                                                                                                  
14:27:57-122875 TRACE      Config: item=theme_type value=Modern default=Standard                                                                                                                        
14:27:57-127457 TRACE      Config: item=keep_incomplete value=False default=True                                                                                                                        
14:27:57-132487 TRACE      Config: item=samples_format value=png default=jpg                                                                                                                            
14:27:57-137318 TRACE      Config: item=save_log_fn value=/workspace/SD_Next/outputs/jobs.json default=                                                                                                 
14:27:57-142397 TRACE      Config: item=directories_filename_pattern value=[model_name] default=[date]                                                                                                  
14:27:57-147390 TRACE      Config: item=samples_filename_pattern value=[datetime<%Y-%m-%d_%H-%M-%S_%f>]-[model_name]-[sampler] default=[seq]-[model_name]-[prompt_words]                                
14:27:57-152898 TRACE      Config: item=live_preview_taesd_layers value=2 default=3                                                                                                                     
14:27:57-157084 TRACE      Config: item=interrogate_return_ranks value=False default=True                                                                                                               
14:27:57-161723 TRACE      Config: item=huggingface_token value=xxxx default=                                                                                          
14:27:57-166373 TRACE      Config: item=lora_in_memory_limit value=3 default=0                                                                                                                          
14:27:57-170363 TRACE      Config: item=extra_networks_styles value=False default=True                                                                                                                  
14:27:57-173851 TRACE      Config: item=gradio_theme value=Default default=black-teal                                                                                                                   
14:27:57-177546 TRACE      Config: item=sd_model_checkpoint value=diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789] default=model.safetensors                                                         
14:27:57-183065 TRACE      Config: item=sd_checkpoint_hash value=None default=                                                                                                                          
14:27:57-188144 TRACE      Config: item=lora_fuse_diffusers value=False default=True                                                                                                                    
14:27:57-193301 TRACE      Config: item=diffusers_dir value=models/diffusers default=models/Diffusers                                                                                                   
14:27:57-198129 TRACE      Config: item=diffusers_offload_min_gpu_memory value=0.5 default=0.25                                                                                                         
14:27:57-213681 TRACE      Config: item=diffusers_offload_max_gpu_memory value=0.9 default=0.7                                                                                                          
14:27:57-218888 TRACE      Config: item=show_progress_type value=TAESD default=Approximate                                                                                                              
14:27:57-223311 TRACE      Config: item=save_to_dirs value=True default=False                                                                                                                           
14:27:57-228311 TRACE      Config: item=clip_skip value=1 default=None                                                                                                                                  
14:27:57-233345 TRACE      Config: item=uni_pc_lower_order_final value=True default=None                                                                                                                
14:27:57-238309 TRACE      Config: item=uni_pc_order value=2 default=None                                                                                                                               
14:27:57-242583 TRACE      Config: item=ckpt_dir value=models/Stable-diffusion default=models/Stable-diffusion                                                                                          
14:27:57-247310 TRACE      Config: item=vae_dir value=models/VAE default=models/VAE                                                                                                                     
14:27:57-251860 TRACE      Config: item=unet_dir value=models/UNET default=models/UNET                                                                                                                  
14:27:57-255544 TRACE      Config: item=te_dir value=models/Text-encoder default=models/Text-encoder                                                                                                    
14:27:57-259842 TRACE      Config: item=lora_dir value=models/Lora default=models/Lora                                                                                                                  
14:27:57-264353 TRACE      Config: item=embeddings_dir value=models/embeddings default=models/embeddings                                                                                                
14:27:57-268849 TRACE      Config: item=hypernetwork_dir value=models/hypernetworks default=models/hypernetworks                                                                                        
14:27:57-273554 TRACE      Config: item=onnx_temp_dir value=models/ONNX/temp default=models/ONNX/temp                                                                                                   
14:27:57-278271 TRACE      Config: item=outdir_txt2img_samples value=outputs/text default=outputs/text                                                                                                  
14:27:57-283148 TRACE      Config: item=outdir_img2img_samples value=outputs/image default=outputs/image                                                                                                
14:27:57-288159 TRACE      Config: item=outdir_control_samples value=outputs/control default=outputs/control                                                                                            
14:27:57-293037 TRACE      Config: item=outdir_extras_samples value=outputs/extras default=outputs/extras                                                                                               
14:27:57-297858 TRACE      Config: item=outdir_init_images value=outputs/init-images default=outputs/init-images                                                                                        
14:27:57-313926 TRACE      Config: item=outdir_txt2img_grids value=outputs/grids default=outputs/grids                                                                                                  
14:27:57-318574 TRACE      Config: item=outdir_img2img_grids value=outputs/grids default=outputs/grids                                                                                                  
14:27:57-323524 TRACE      Config: item=outdir_control_grids value=outputs/grids default=outputs/grids                                                                                                  
14:27:57-328066 TRACE      Config: item=outdir_save value=outputs/save default=outputs/save                                                                                                             
14:27:57-332653 TRACE      Config: item=outdir_video value=outputs/video default=outputs/video                                                                                                          
14:27:57-337309 TRACE      Config: item=yolo_dir value=models/yolo default=models/yolo                                                                                                                  
14:27:57-341773 TRACE      Config: item=wildcards_dir value=models/wildcards default=models/wildcards                                                                                                   
14:27:57-345505 TRACE      Config: item=hfcache_dir value=/workspace/SD_Next/models/huggingface default=/root/.cache/huggingface/hub                                                                    
14:27:57-360940 DEBUG    Save: file="config.json" json=43 bytes=1896 time=0.011                                                                                                                         
14:27:57-366645 TRACE    Settings save: {'sdp_options': ['Flash attention', 'Memory attention', 'Math attention'], 'xformers_options': ['Flash attention'], 'ipex_optimize': [], 'openvino_devices': [],
                         'bnb_quantization': [], 'optimum_quanto_weights': [], 'torchao_quantization': [], 'nncf_compress_weights': [], 'nncf_quantize': [], 'cuda_compile': [], 'styles_dir':          
                         'models/styles', 'quicksettings_list': ['sd_model_checkpoint', 'sd_model_refiner', 'sd_unet', 'sd_vae', 'sd_text_encoder', 'show_progress_type'], 'show_samplers': [],         
                         'postprocessing_enable_in_main_ui': [], 'postprocessing_operation_order': [], 'detailer_models': ['face-yolo8n', 'eyes-full-v1', 'hand_yolov8n'],                              
                         'interrogate_clip_skip_categories': ['artists', 'movements', 'flavors'], 'extra_networks': ['All'], 'diffusers_version': '03bcf5aefef13a064c34b605e489c0730052cca8',           
                         'disabled_extensions': [], 'detailer_conf': 0.5, 'detailer_max': 4, 'theme_type': 'Modern', 'keep_incomplete': False, 'samples_format': 'png', 'save_log_fn':                  
                         '/workspace/SD_Next/outputs/jobs.json', 'directories_filename_pattern': '[model_name]', 'samples_filename_pattern': '[datetime<%Y-%m-%d_%H-%M-%S_%f>]-[model_name]-[sampler]', 
                         'live_preview_taesd_layers': 2, 'interrogate_return_ranks': False, 'huggingface_token': 'xxx', 'lora_in_memory_limit': 3,                    
                         'extra_networks_styles': False, 'gradio_theme': 'Default', 'sd_model_checkpoint': 'diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789]', 'sd_checkpoint_hash': None,           
                         'lora_fuse_diffusers': False, 'diffusers_dir': 'models/diffusers', 'diffusers_offload_min_gpu_memory': 0.5, 'diffusers_offload_max_gpu_memory': 0.9, 'show_progress_type':     
                         'TAESD', 'save_to_dirs': True, 'hfcache_dir': '/workspace/SD_Next/models/huggingface'}                                                                                         
14:27:59-578090 DEBUG    Server: alive=True requests=207 memory=1.17/503.51 status='running' task='Load' timestamp='20250112142622' id='' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=126       
                         elapsed=97.23 eta=None progress=0

@IowaSovereign
Copy link
Author

IowaSovereign commented Jan 12, 2025 •
edited by vladmandic
Loading

This is the log of the restart, and I can confirm that the line gets removed from config at the very last moment, after the model is loaded. There is some sort of save operation, and also there's a curious line:
Config: item=hfcache_dir value=/workspace/SD_Next/models/huggingface default=/workspace/SD_Next/models/huggingface

Which is wrong as the default should still stay as it was, as root something.

14:29:30-130119 INFO     Starting SD.Next                                                                                                                                                               
14:29:30-144602 INFO     Logger: file="/workspace/SD_Next/sdnext.log" level=DEBUG size=64 mode=create                                                                                                   
14:29:30-151097 INFO     Python: version=3.10.12 platform=Linux bin="/workspace/SD_Next/venv/bin/python3" venv="/workspace/SD_Next/venv"                                                                
14:29:30-585490 INFO     Version: app=sd.next updated=2025-01-11 hash=b4575ad8 branch=dev url=https://github.com/vladmandic/automatic.git/tree/dev ui=dev                                               
14:29:31-195256 INFO     Repository latest available 3f36089142d6f71b486ab39f316405946aee8752 2025-01-02T21:54:48Z                                                                                      
14:29:31-241191 INFO     Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.5.0-35-generic python=3.10.12 docker=False                                                                             
14:29:31-247592 DEBUG    Packages: venv=venv site=['venv/lib/python3.10/site-packages']                                                                                                                 
14:29:31-253776 INFO     Args: ['--listen', '--port', '3031', '--debug', '--safe']                                                                                                                      
14:29:31-260333 DEBUG    Setting environment tuning                                                                                                                                                     
14:29:31-264081 DEBUG    Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"                                                                                                     
14:29:31-291859 DEBUG    Torch overrides: cuda=False rocm=False ipex=False directml=False openvino=False zluda=False                                                                                    
14:29:31-298191 INFO     CUDA: nVidia toolkit detected                                                                                                                                                  
14:29:32-134762 WARNING  Deleted files: ['models', 'models/.placeholder', 'models/Reference/Alpha-VLLM--Lumina-Next-SFT-diffusers.jpg', 'models/Reference/DeepFloyd--IF-I-M-v1.0.jpg',                  
                         'models/Reference/Efficient-Large-Model--Sana_1600M_1024px_diffusers.jpg', 'models/Reference/IDKiro--sdxs-512-dreamshaper.jpg',                                                
                         'models/Reference/Kwai-Kolors--Kolors-diffusers.jpg', 'models/Reference/MeissonFlow--Meissonic.jpg', 'models/Reference/PixArt-alpha--PixArt-Sigma-XL-2-1024-MS.jpg',           
                         'models/Reference/PixArt-alpha--PixArt-XL-2-1024-MS.jpg', 'models/Reference/PixArt-alpha--PixArt-XL-2-512x512.jpg',                                                            
                         'models/Reference/PixArt-alpha--pixart_sigma_sdxlvae_T5_diffusers.jpg', 'models/Reference/SDXL-Flash_Mini.jpg', 'models/Reference/Shitao--OmniGen-v1.jpg',                     
                         'models/Reference/SimianLuo--LCM_Dreamshaper_v7.jpg', 'models/Reference/THUDM--CogView3-Plus-3B.jpg', 'models/Reference/TempestV0.1-Artistic.jpg',                             
                         'models/Reference/Tencent-Hunyuan--HunyuanDiT-v1.2-Diffusers.jpg', 'models/Reference/XCLiu--instaflow_0_9B_from_sd_1_5.jpg', 'models/Reference/amused--amused-256.jpg',        
                         'models/Reference/amused--amused-512.jpg', 'models/Reference/black-forest-labs--FLUX.1-dev.jpg', 'models/Reference/black-forest-labs--FLUX.1-schnell.jpg',                     
                         'models/Reference/damo-vilab--text-to-video-ms-1.7b.jpg', 'models/Reference/dreamshaperXL_v21TurboDPMSDE.jpg', 'models/Reference/dreamshaper_8.jpg',                           
                         'models/Reference/etri-vilab--koala-700m-llava-cap.jpg', 'models/Reference/fal--AuraFlow-v0.3.jpg', 'models/Reference/juggernautXL_v9Rdphoto2Lightning.jpg',                   
                         'models/Reference/juggernautXL_v9Rundiffusionphoto2.jpg', 'models/Reference/juggernaut_reborn.jpg', 'models/Reference/kandinsky-community--kandinsky-2-1.jpg',                 
                         'models/Reference/kandinsky-community--kandinsky-2-2-decoder.jpg', 'models/Reference/kandinsky-community--kandinsky-3.jpg',                                                    
                         'models/Reference/latent-consistency--lcm-sdxl.jpg', 'models/Reference/playgroundai--playground-v1.jpg', 'models/Reference/playgroundai--playground-v2-1024px-aesthetic.jpg',  
                         'models/Reference/playgroundai--playground-v2-256px-base.jpg', 'models/Reference/playgroundai--playground-v2-512px-base.jpg',                                                  
                         'models/Reference/playgroundai--playground-v2.5-1024px-aesthetic.jpg', 'models/Reference/salesforce--blipdiffusion.jpg', 'models/Reference/sd_xl_base_1.0.jpg',                
                         'models/Reference/sdxl_turbo.jpg', 'models/Reference/segmind--SSD-1B.jpg', 'models/Reference/segmind--SegMoE-2x1-v0.jpg', 'models/Reference/segmind--SegMoE-4x2-v0.jpg',       
                         'models/Reference/segmind--SegMoE-SD-4x2-v0.jpg', 'models/Reference/segmind--Segmind-Vega.jpg', 'models/Reference/segmind--tiny-sd.jpg',                                       
                         'models/Reference/shuttleai--shuttle-3-diffusion.jpg', 'models/Reference/stabilityai--stable-cascade.jpg', 'models/Reference/stabilityai--stable-diffusion-2-1-base.jpg',      
                         'models/Reference/stabilityai--stable-diffusion-2-1.jpg', 'models/Reference/stabilityai--stable-diffusion-3.jpg', 'models/Reference/stabilityai--stable-diffusion-3_5.jpg',    
                         'models/Reference/stabilityai--stable-video-diffusion-img2vid-xt.jpg', 'models/Reference/stabilityai--stable-video-diffusion-img2vid.jpg',                                     
                         'models/Reference/thu-ml--unidiffuser-v1.jpg', 'models/Reference/v1-5-pruned-fp16-emaonly.jpg', 'models/Reference/warp-ai--wuerstchen.jpg', 'models/VAE-approx/model.pt',      
                         'models/karlo/ViT-L-14_stats.th']                                                                                                                                              
14:29:32-218198 WARNING  Modified files: ['models/Reference/Alpha-VLLM--Lumina-Next-SFT-diffusers.jpg', 'models/Reference/DeepFloyd--IF-I-M-v1.0.jpg',                                                  
                         'models/Reference/IDKiro--sdxs-512-dreamshaper.jpg', 'models/Reference/Kwai-Kolors--Kolors-diffusers.jpg', 'models/Reference/PixArt-alpha--PixArt-Sigma-XL-2-1024-MS.jpg',     
                         'models/Reference/PixArt-alpha--PixArt-XL-2-1024-MS.jpg', 'models/Reference/PixArt-alpha--PixArt-XL-2-512x512.jpg',                                                            
                         'models/Reference/PixArt-alpha--pixart_sigma_sdxlvae_T5_diffusers.jpg', 'models/Reference/SDXL-Flash_Mini.jpg', 'models/Reference/SimianLuo--LCM_Dreamshaper_v7.jpg',          
                         'models/Reference/TempestV0.1-Artistic.jpg', 'models/Reference/Tencent-Hunyuan--HunyuanDiT-v1.2-Diffusers.jpg', 'models/Reference/XCLiu--instaflow_0_9B_from_sd_1_5.jpg',      
                         'models/Reference/amused--amused-256.jpg', 'models/Reference/amused--amused-512.jpg', 'models/Reference/black-forest-labs--FLUX.1-dev.jpg',                                    
                         'models/Reference/black-forest-labs--FLUX.1-schnell.jpg', 'models/Reference/damo-vilab--text-to-video-ms-1.7b.jpg', 'models/Reference/dreamshaperXL_v21TurboDPMSDE.jpg',       
                         'models/Reference/dreamshaper_8.jpg', 'models/Reference/etri-vilab--koala-700m-llava-cap.jpg', 'models/Reference/fal--AuraFlow-v0.3.jpg',                                      
                         'models/Reference/juggernautXL_v9Rdphoto2Lightning.jpg', 'models/Reference/juggernautXL_v9Rundiffusionphoto2.jpg', 'models/Reference/juggernaut_reborn.jpg',                   
                         'models/Reference/kandinsky-community--kandinsky-2-1.jpg', 'models/Reference/kandinsky-community--kandinsky-2-2-decoder.jpg',                                                  
                         'models/Reference/kandinsky-community--kandinsky-3.jpg', 'models/Reference/latent-consistency--lcm-sdxl.jpg', 'models/Reference/playgroundai--playground-v1.jpg',              
                         'models/Reference/playgroundai--playground-v2-1024px-aesthetic.jpg', 'models/Reference/playgroundai--playground-v2-256px-base.jpg',                                            
                         'models/Reference/playgroundai--playground-v2-512px-base.jpg', 'models/Reference/playgroundai--playground-v2.5-1024px-aesthetic.jpg',                                          
                         'models/Reference/salesforce--blipdiffusion.jpg', 'models/Reference/sd_xl_base_1.0.jpg', 'models/Reference/sdxl_turbo.jpg', 'models/Reference/segmind--SSD-1B.jpg',            
                         'models/Reference/segmind--SegMoE-2x1-v0.jpg', 'models/Reference/segmind--SegMoE-4x2-v0.jpg', 'models/Reference/segmind--SegMoE-SD-4x2-v0.jpg',                                
                         'models/Reference/segmind--Segmind-Vega.jpg', 'models/Reference/segmind--tiny-sd.jpg', 'models/Reference/stabilityai--stable-cascade.jpg',                                     
                         'models/Reference/stabilityai--stable-diffusion-2-1-base.jpg', 'models/Reference/stabilityai--stable-diffusion-2-1.jpg',                                                       
                         'models/Reference/stabilityai--stable-diffusion-3.jpg', 'models/Reference/stabilityai--stable-video-diffusion-img2vid-xt.jpg',                                                 
                         'models/Reference/stabilityai--stable-video-diffusion-img2vid.jpg', 'models/Reference/thu-ml--unidiffuser-v1.jpg', 'models/Reference/v1-5-pruned-fp16-emaonly.jpg',            
                         'models/Reference/warp-ai--wuerstchen.jpg', 'models/VAE-approx/model.pt', 'models/karlo/ViT-L-14_stats.th']                                                                    
14:29:33-056811 INFO     Install: verifying requirements                                                                                                                                                
14:29:33-077504 INFO     Verifying packages                                                                                                                                                             
14:29:33-161198 DEBUG    Timestamp repository update time: Sat Jan 11 23:51:33 2025                                                                                                                     
14:29:33-165263 INFO     Startup: standard                                                                                                                                                              
14:29:33-168658 INFO     Verifying submodules                                                                                                                                                           
14:29:37-884384 DEBUG    Git submodule: extensions-builtin/sd-extension-chainner / main                                                                                                                 
14:29:38-262880 DEBUG    Git submodule: extensions-builtin/sd-extension-system-info / main                                                                                                              
14:29:38-562615 DEBUG    Git submodule: extensions-builtin/sd-webui-agent-scheduler / main                                                                                                              
14:29:38-857802 DEBUG    Git submodule: extensions-builtin/sdnext-modernui / dev                                                                                                                        
14:29:39-066686 DEBUG    Git submodule: extensions-builtin/stable-diffusion-webui-rembg / master                                                                                                        
14:29:39-235209 DEBUG    Git submodule: modules/k-diffusion / master                                                                                                                                    
14:29:39-387824 DEBUG    Git submodule: wiki / master                                                                                                                                                   
14:29:39-498975 DEBUG    Register paths                                                                                                                                                                 
14:29:40-084051 DEBUG    Installed packages: 282                                                                                                                                                        
14:29:40-088954 DEBUG    Extensions all: ['sdnext-modernui', 'stable-diffusion-webui-rembg', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-extension-chainner', 'Lora']                   
14:29:40-623671 DEBUG    Extension installer: /workspace/SD_Next/extensions-builtin/stable-diffusion-webui-rembg/install.py                                                                             
14:30:00-422592 DEBUG    Extension installer: /workspace/SD_Next/extensions-builtin/sd-webui-agent-scheduler/install.py                                                                                 
14:30:08-659292 INFO     Extensions enabled: ['sdnext-modernui', 'stable-diffusion-webui-rembg', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-extension-chainner', 'Lora']               
14:30:08-663700 INFO     Install: verifying requirements                                                                                                                                                
14:30:08-668400 DEBUG    Setup complete without errors: 1736692209                                                                                                                                      
14:30:08-672347 INFO     Running in safe mode without user extensions                                                                                                                                   
14:30:08-689079 DEBUG    Extension preload: {'extensions-builtin': 0.01}                                                                                                                                
14:30:08-694997 INFO     Command line args: ['--listen', '--port', '3031', '--debug', '--safe'] listen=True port=3031 safe=True debug=True args=[]                                                      
14:30:08-700177 DEBUG    Env flags: ['SD_CONFIG_DEBUG=true=true']                                                                                                                                       
14:30:08-717102 DEBUG    Starting module: <module 'webui' from '/workspace/SD_Next/webui.py'>                                                                                                           
14:30:32-837942 INFO     Device detect: memory=44.3 default=balanced                                                                                                                                    
14:30:32-863733 DEBUG    Read: file="config.json" json=43 bytes=1896 time=0.001 fn=<module>:load                                                                                                        
14:30:33-939271 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda device=cuda attention="Scaled-Dot-Product" mode=no_grad                                                                         
14:30:33-945580 DEBUG    Read: file="html/reference.json" json=62 bytes=32535 time=0.001 fn=_call_with_frames_removed:<module>                                                                          
14:30:34-029149 INFO     Torch parameters: backend=cuda device=cuda config=Auto dtype=torch.bfloat16 context=no_grad nohalf=False nohalfvae=False upcast=False deterministic=False fp16=pass bf16=pass  
                         optimization="Scaled-Dot-Product"                                                                                                                                              
14:30:34-985519 DEBUG    ONNX: version=1.19.0 provider=CUDAExecutionProvider, available=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']                                  
14:30:35-627878 DEBUG    ONNX failed to initialize XL pipelines: module 'optimum.onnxruntime.modeling_diffusion' has no attribute 'ORTPipelinePart'                                                     
14:30:35-696023 INFO     Device: device=NVIDIA A40 n=1 arch=sm_90 capability=(8, 6) cuda=12.1 cudnn=90100 driver=550.127.05                                                                             
14:30:44-413440 DEBUG    System: cores=96 affinity=96 threads=48                                                                                                                                        
14:30:44-419513 INFO     Torch: torch==2.4.0+cu121 torchvision==0.19.0+cu121                                                                                                                            
14:30:44-422240 INFO     Packages: diffusers==0.33.0.dev0 transformers==4.47.1 accelerate==1.2.1 gradio==3.43.2                                                                                         
14:30:44-752949 DEBUG    Entering start sequence                                                                                                                                                        
14:30:44-772506 DEBUG    Initializing                                                                                                                                                                   
14:30:44-782560 DEBUG    Read: file="metadata.json" json=246 bytes=593555 time=0.007 fn=initialize:init_metadata                                                                                        
14:30:44-788768 DEBUG    Read: file="cache.json" json=2 bytes=13730 time=0.001 fn=initialize:init_cache                                                                                                 
14:30:44-792632 DEBUG    Huggingface cache: path="/workspace/SD_Next/models/huggingface"                                                                                                                
14:30:44-955374 INFO     Available VAEs: path="models/VAE" items=11                                                                                                                                     
14:30:45-019590 INFO     Available UNets: path="models/UNET" items=37                                                                                                                                   
14:30:45-031907 INFO     Available TEs: path="models/Text-encoder" items=9                                                                                                                              
14:30:45-194332 INFO     Available Models: items=42 safetensors="models/Stable-diffusion":37 diffusers="models/diffusers":5 time=0.16                                                                   
14:30:45-317460 INFO     Available LoRAs: path="models/Lora" items=84 folders=8 time=0.07                                                                                                               
14:30:45-343598 INFO     Available Styles: folder="models/styles" items=20 time=0.02                                                                                                                    
14:30:45-693172 INFO     Available Yolo: path="models/yolo" items=7 downloaded=4                                                                                                                        
14:30:45-697865 DEBUG    Extensions: disabled=['Lora', 'sd-extension-chainner', 'sd-webui-agent-scheduler', 'stable-diffusion-webui-rembg']                                                             
14:30:45-699767 INFO     Load extensions                                                                                                                                                                
14:30:47-340147 DEBUG    Extensions init time: total=1.63 pulid_ext.py=1.32                                                                                                                             
14:30:47-392484 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2640 time=0.002 fn=__init__:__init__                                                                                             
14:30:47-418278 DEBUG    Upscaler type=ESRGAN folder="models/ESRGAN" model="DAT-Helaman-Nomos-4x" path="models/ESRGAN/DAT-Helaman-Nomos-4x.pth"                                                         
14:30:47-446121 INFO     Available Upscalers: items=29 downloaded=6 user=1 time=0.09 types=['None', 'Lanczos', 'Nearest', 'ESRGAN', 'SwinIR', 'AuraSR', 'SD', 'SCUNet', 'RealESRGAN']                   
14:30:47-475901 DEBUG    UI start sequence                                                                                                                                                              
14:30:47-482545 INFO     UI theme: type=Modern name="Default" available=34                                                                                                                              
14:30:47-539381 DEBUG    UI theme: css="extensions-builtin/sdnext-modernui/themes/Default.css" base="base.css" user="None"                                                                              
14:30:47-550265 DEBUG    UI initialize: txt2img                                                                                                                                                         
14:30:48-292931 DEBUG    Networks: page='model' items=103 subfolders=7 tab=txt2img folders=['models/Stable-diffusion', 'models/diffusers', 'models/Reference'] list=0.60 thumb=0.23 desc=0.28 info=0.81 
                         workers=8                                                                                                                                                                      
14:30:48-457157 DEBUG    Networks: page='lora' items=84 subfolders=8 tab=txt2img folders=['models/Lora'] list=0.57 thumb=0.23 desc=0.12 info=1.29 workers=8                                             
14:30:48-488339 DEBUG    Networks: page='style' items=20 subfolders=2 tab=txt2img folders=['models/styles', 'html'] list=0.33 thumb=0.30 desc=0.00 info=0.00 workers=8                                  
14:30:48-499397 DEBUG    Networks: page='embedding' items=0 subfolders=0 tab=txt2img folders=['models/embeddings'] list=0.02 thumb=0.01 desc=0.00 info=0.00 workers=8                                   
14:30:48-519226 DEBUG    Networks: page='vae' items=11 subfolders=0 tab=txt2img folders=['models/VAE'] list=0.42 thumb=0.35 desc=0.01 info=0.02 workers=8                                               
14:30:48-526399 DEBUG    Networks: page='history' items=0 subfolders=0 tab=txt2img folders=[] list=0.00 thumb=0.00 desc=0.00 info=0.00 workers=8                                                        
14:30:49-052605 DEBUG    UI initialize: img2img                                                                                                                                                         
14:30:49-329004 DEBUG    UI initialize: control models=models/control                                                                                                                                   
14:30:50-422033 DEBUG    Read: file="ui-config.json" json=27 bytes=1596 time=0.002 fn=__init__:read_from_file                                                                                           
14:30:51-724561 DEBUG    Extension list: processed=397 installed=6 enabled=2 disabled=4 visible=397 hidden=0                                                                                            
14:30:52-208498 DEBUG    Root paths: ['/workspace/SD_Next']                                                                                                                                             
14:30:52-396041 INFO     Local URL: http://localhost:3031/                                                                                                                                              
14:30:52-425253 DEBUG    API middleware: [<class 'starlette.middleware.base.BaseHTTPMiddleware'>, <class 'starlette.middleware.gzip.GZipMiddleware'>]                                                   
14:30:52-441088 DEBUG    API initialize                                                                                                                                                                 
14:30:52-650730 DEBUG    SD-System-Info: benchmark data loaded: /workspace/SD_Next/extensions-builtin/sd-extension-system-info/scripts/benchmark-data-local.json                                        
14:30:52-724789 DEBUG    Scripts setup: time=0.662 ['K-Diffusion Samplers:0.245', 'XYZ Grid:0.061', 'IP Adapters:0.057', 'Face: Multiple ID Transfers:0.034', 'FreeScale: Tuning-Free Scale             
                         Fusion:0.017', 'Video: LTX Video:0.015', 'Video: CogVideoX:0.015', 'Video: AnimateDiff:0.013', 'ConsiStory: Consistent Image Generation:0.013', 'Video: VGen                   
                         Image-to-Video:0.012', 'PuLID: ID Customization:0.011', 'Ctrl-X: Controlling Structure and Appearance:0.011', 'LUT Color grading:0.011', 'Style Aligned Image Generation:0.01']
14:30:52-747350 DEBUG    Model metadata: file="metadata.json" no changes                                                                                                                                
14:30:52-750258 DEBUG    Model requested: fn=run:<lambda>                                                                                                                                               
14:30:52-752991 INFO     Load model: select="diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789]"                                                                                                       
14:30:52-766217 DEBUG    Load model: type=FLUX model="diffusers/black-forest-labs/FLUX.1-dev" repo="black-forest-labs/FLUX.1-dev" unet="None" te="None" vae="Automatic" quant=none offload=balanced     
                         dtype=torch.bfloat16                                                                                                                                                           
14:30:52-847671 INFO     HF login: token="/root/.cache/huggingface/token"                                                                                                                               
14:30:53-131790 DEBUG    GC: current={'gpu': 0.56, 'ram': 1.01, 'oom': 0} prev={'gpu': 0.57, 'ram': 1.01} load={'gpu': 1, 'ram': 0} gc={'gpu': 0.01, 'py': 5159} fn=load_diffuser_force:load_flux       
                         why=force time=0.28                                                                                                                                                            
14:30:53-134555 DEBUG    Load model: type=FLUX cls=FluxPipeline preloaded=[] revision=None                                                                                                              
Diffusers  3.40it/s █████████████ 100% 2/2 00:00 00:00 Loading checkpoint shards
Diffusers  2.57it/s ████████ 100% 7/7 00:02 00:00 Loading pipeline components...
14:30:56-077848 DEBUG    Setting model: component=VAE slicing=True                                                                                                                                      
14:30:56-080507 DEBUG    Setting model: attention="Scaled-Dot-Product"                                                                                                                                  
14:30:56-099639 INFO     Offload: type=balanced op=init watermark=0.5-0.9 gpu=22.170-39.906000000000006:44.34 cpu=503.510 limit=0.00                                                                    
14:30:56-128869 DEBUG    Model module=transformer type=FluxTransformer2DModel dtype=torch.bfloat16 quant=None params=11.084 size=22.168                                                                 
14:30:56-134177 DEBUG    Model module=text_encoder_2 type=T5EncoderModel dtype=torch.bfloat16 quant=None params=4.435 size=8.870                                                                        
14:30:56-137916 DEBUG    Model module=text_encoder type=CLIPTextModel dtype=torch.bfloat16 quant=None params=0.115 size=0.229                                                                           
14:30:56-140879 DEBUG    Model module=vae type=AutoencoderKL dtype=torch.bfloat16 quant=None params=0.078 size=0.156                                                                                    
14:30:56-143332 INFO     Model class=FluxPipeline modules=4 size=31.424                                                                                                                                 
14:30:56-148237 INFO     Load model: time=total=3.39 load=3.32 native=1024 memory={'ram': {'used': 1.13, 'total': 503.51}, 'gpu': {'used': 0.56, 'total': 44.34}, 'retries': 0, 'oom': 0}               
14:30:56-155072 DEBUG    Settings: user                                                                                                                                                                 
14:30:56-156059 DEBUG    Script init: ['system-info.py:app_started=0.09']                                                                                                                               
14:30:56-159708 TRACE      Config: item=sdp_options value=['Flash attention', 'Memory attention', 'Math attention'] default=['Flash attention', 'Memory attention', 'Math attention']                   
14:30:56-162610 INFO     Startup time: total=87.84 torch=76.08 launch=40.69 installer=40.40 checkpoint=3.43 extensions=1.63 ui-networks=1.30 ui-extensions=0.91 ui-models=0.50 ui-txt2img=0.49          
                         ui-settings=0.45 ui-defaults=0.37 detailer=0.35 libraries=0.35 ui-control=0.32 api=0.20 samplers=0.19 ui-img2img=0.18 models=0.16 lora=0.12 upscalers=0.12 app-started=0.09    
                         ui-extras=0.09 unet=0.06                                                                                                                                                       
14:30:56-170442 TRACE      Config: item=xformers_options value=['Flash attention'] default=['Flash attention']                                                                                          
14:30:56-179551 TRACE      Config: item=ipex_optimize value=[] default=[]                                                                                                                               
14:30:56-182000 TRACE      Config: item=openvino_devices value=[] default=[]                                                                                                                            
14:30:56-183980 TRACE      Config: item=bnb_quantization value=[] default=[]                                                                                                                            
14:30:56-185988 TRACE      Config: item=optimum_quanto_weights value=[] default=[]                                                                                                                      
14:30:56-188015 TRACE      Config: item=torchao_quantization value=[] default=[]                                                                                                                        
14:30:56-189986 TRACE      Config: item=nncf_compress_weights value=[] default=[]                                                                                                                       
14:30:56-192318 TRACE      Config: item=nncf_quantize value=[] default=[]                                                                                                                               
14:30:56-194297 TRACE      Config: item=cuda_compile value=[] default=[]                                                                                                                                
14:30:56-196869 TRACE      Config: item=styles_dir value=models/styles default=styles.csv                                                                                                               
14:30:56-198840 TRACE      Config: item=quicksettings_list value=['sd_model_checkpoint', 'sd_model_refiner', 'sd_unet', 'sd_vae', 'sd_text_encoder', 'show_progress_type']                              
                         default=['sd_model_checkpoint']                                                                                                                                                
14:30:56-211009 TRACE      Config: item=show_samplers value=[] default=[]                                                                                                                               
14:30:56-213622 TRACE      Config: item=postprocessing_enable_in_main_ui value=[] default=[]                                                                                                            
14:30:56-216682 TRACE      Config: item=postprocessing_operation_order value=[] default=[]                                                                                                              
14:30:56-218968 TRACE      Config: item=detailer_models value=['face-yolo8n', 'eyes-full-v1', 'hand_yolov8n'] default=['face-yolo8n']                                                                   
14:30:56-221971 TRACE      Config: item=interrogate_clip_skip_categories value=['artists', 'movements', 'flavors'] default=['artists', 'movements', 'flavors']                                          
14:30:56-224518 TRACE      Config: item=extra_networks value=['All'] default=['All']                                                                                                                    
14:30:56-227353 TRACE      Config: item=diffusers_version value=03bcf5aefef13a064c34b605e489c0730052cca8 default=                                                                                       
14:30:56-230144 TRACE      Config: item=disabled_extensions value=[] default=[]                                                                                                                         
14:30:56-232780 TRACE      Config: item=detailer_conf value=0.5 default=0.6                                                                                                                             
14:30:56-235569 TRACE      Config: item=detailer_max value=4 default=2                                                                                                                                  
14:30:56-238117 TRACE      Config: item=theme_type value=Modern default=Standard                                                                                                                        
14:30:56-240664 TRACE      Config: item=keep_incomplete value=False default=True                                                                                                                        
14:30:56-243281 TRACE      Config: item=samples_format value=png default=jpg                                                                                                                            
14:30:56-245819 TRACE      Config: item=save_log_fn value=/workspace/SD_Next/outputs/jobs.json default=                                                                                                 
14:30:56-248426 TRACE      Config: item=directories_filename_pattern value=[model_name] default=[date]                                                                                                  
14:30:56-251947 TRACE      Config: item=samples_filename_pattern value=[datetime<%Y-%m-%d_%H-%M-%S_%f>]-[model_name]-[sampler] default=[seq]-[model_name]-[prompt_words]                                
14:30:56-255451 TRACE      Config: item=live_preview_taesd_layers value=2 default=3                                                                                                                     
14:30:56-258225 TRACE      Config: item=interrogate_return_ranks value=False default=True                                                                                                               
14:30:56-261070 TRACE      Config: item=huggingface_token value=xxx default=                                                                                          
14:30:56-263844 TRACE      Config: item=lora_in_memory_limit value=3 default=0                                                                                                                          
14:30:56-267228 TRACE      Config: item=extra_networks_styles value=False default=True                                                                                                                  
14:30:56-269855 TRACE      Config: item=gradio_theme value=Default default=black-teal                                                                                                                   
14:30:56-272504 TRACE      Config: item=sd_model_checkpoint value=diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789] default=model.safetensors                                                         
14:30:56-275483 TRACE      Config: item=sd_checkpoint_hash value=None default=                                                                                                                          
14:30:56-278125 TRACE      Config: item=lora_fuse_diffusers value=False default=True                                                                                                                    
14:30:56-281104 TRACE      Config: item=diffusers_dir value=models/diffusers default=models/Diffusers                                                                                                   
14:30:56-283744 TRACE      Config: item=diffusers_offload_min_gpu_memory value=0.5 default=0.25                                                                                                         
14:30:56-287019 TRACE      Config: item=diffusers_offload_max_gpu_memory value=0.9 default=0.7                                                                                                          
14:30:56-289849 TRACE      Config: item=show_progress_type value=TAESD default=Approximate                                                                                                              
14:30:56-292490 TRACE      Config: item=save_to_dirs value=True default=False                                                                                                                           
14:30:56-295179 TRACE      Config: item=hfcache_dir value=/workspace/SD_Next/models/huggingface default=/workspace/SD_Next/models/huggingface                                                           
14:30:56-297865 TRACE      Config: item=clip_skip value=1 default=None                                                                                                                                  
14:30:56-300412 TRACE      Config: item=uni_pc_lower_order_final value=True default=None                                                                                                                
14:30:56-313482 TRACE      Config: item=uni_pc_order value=2 default=None                                                                                                                               
14:30:56-317079 TRACE      Config: item=ckpt_dir value=models/Stable-diffusion default=models/Stable-diffusion                                                                                          
14:30:56-320067 TRACE      Config: item=vae_dir value=models/VAE default=models/VAE                                                                                                                     
14:30:56-323112 TRACE      Config: item=unet_dir value=models/UNET default=models/UNET                                                                                                                  
14:30:56-325990 TRACE      Config: item=te_dir value=models/Text-encoder default=models/Text-encoder                                                                                                    
14:30:56-328845 TRACE      Config: item=lora_dir value=models/Lora default=models/Lora                                                                                                                  
14:30:56-332385 TRACE      Config: item=embeddings_dir value=models/embeddings default=models/embeddings                                                                                                
14:30:56-335392 TRACE      Config: item=hypernetwork_dir value=models/hypernetworks default=models/hypernetworks                                                                                        
14:30:56-338275 TRACE      Config: item=onnx_temp_dir value=models/ONNX/temp default=models/ONNX/temp                                                                                                   
14:30:56-341725 TRACE      Config: item=outdir_txt2img_samples value=outputs/text default=outputs/text                                                                                                  
14:30:56-344612 TRACE      Config: item=outdir_img2img_samples value=outputs/image default=outputs/image                                                                                                
14:30:56-347627 TRACE      Config: item=outdir_control_samples value=outputs/control default=outputs/control                                                                                            
14:30:56-350525 TRACE      Config: item=outdir_extras_samples value=outputs/extras default=outputs/extras                                                                                               
14:30:56-353451 TRACE      Config: item=outdir_init_images value=outputs/init-images default=outputs/init-images                                                                                        
14:30:56-356907 TRACE      Config: item=outdir_txt2img_grids value=outputs/grids default=outputs/grids                                                                                                  
14:30:56-359787 TRACE      Config: item=outdir_img2img_grids value=outputs/grids default=outputs/grids                                                                                                  
14:30:56-362771 TRACE      Config: item=outdir_control_grids value=outputs/grids default=outputs/grids                                                                                                  
14:30:56-365720 TRACE      Config: item=outdir_save value=outputs/save default=outputs/save                                                                                                             
14:30:56-368724 TRACE      Config: item=outdir_video value=outputs/video default=outputs/video                                                                                                          
14:30:56-372388 TRACE      Config: item=yolo_dir value=models/yolo default=models/yolo                                                                                                                  
14:30:56-375385 TRACE      Config: item=wildcards_dir value=models/wildcards default=models/wildcards                                                                                                   
14:30:56-382803 DEBUG    Save: file="config.json" json=42 bytes=1838 time=0.004                                                                                                                         
14:30:56-387802 TRACE    Settings save: {'sdp_options': ['Flash attention', 'Memory attention', 'Math attention'], 'xformers_options': ['Flash attention'], 'ipex_optimize': [], 'openvino_devices': [],
                         'bnb_quantization': [], 'optimum_quanto_weights': [], 'torchao_quantization': [], 'nncf_compress_weights': [], 'nncf_quantize': [], 'cuda_compile': [], 'styles_dir':          
                         'models/styles', 'quicksettings_list': ['sd_model_checkpoint', 'sd_model_refiner', 'sd_unet', 'sd_vae', 'sd_text_encoder', 'show_progress_type'], 'show_samplers': [],         
                         'postprocessing_enable_in_main_ui': [], 'postprocessing_operation_order': [], 'detailer_models': ['face-yolo8n', 'eyes-full-v1', 'hand_yolov8n'],                              
                         'interrogate_clip_skip_categories': ['artists', 'movements', 'flavors'], 'extra_networks': ['All'], 'diffusers_version': '03bcf5aefef13a064c34b605e489c0730052cca8',           
                         'disabled_extensions': [], 'detailer_conf': 0.5, 'detailer_max': 4, 'theme_type': 'Modern', 'keep_incomplete': False, 'samples_format': 'png', 'save_log_fn':                  
                         '/workspace/SD_Next/outputs/jobs.json', 'directories_filename_pattern': '[model_name]', 'samples_filename_pattern': '[datetime<%Y-%m-%d_%H-%M-%S_%f>]-[model_name]-[sampler]', 
                         'live_preview_taesd_layers': 2, 'interrogate_return_ranks': False, 'huggingface_token': 'xxx', 'lora_in_memory_limit': 3,                    
                         'extra_networks_styles': False, 'gradio_theme': 'Default', 'sd_model_checkpoint': 'diffusers/black-forest-labs/FLUX.1-dev [0ef5fff789]', 'sd_checkpoint_hash': None,           
                         'lora_fuse_diffusers': False, 'diffusers_dir': 'models/diffusers', 'diffusers_offload_min_gpu_memory': 0.5, 'diffusers_offload_max_gpu_memory': 0.9, 'show_progress_type':     
                         'TAESD', 'save_to_dirs': True}

@IowaSovereign
Copy link
Author

And output of the env command as requested.

SHELL=/bin/bash
NV_LIBCUBLAS_VERSION=12.1.3.1-1
NVIDIA_VISIBLE_DEVICES=GPU-447821c5-29e2-6d72-e1c7-ee730e9f3bb7
NV_NVML_DEV_VERSION=12.1.105-1
NV_CUDNN_PACKAGE_NAME=libcudnn8
PYTHONUNBUFFERED=1
TEMPLATE_VERSION=latest
NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
HOSTNAME=6f45b2ea06d8
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
NV_NVTX_VERSION=12.1.105-1
NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
NV_LIBCUSPARSE_VERSION=12.1.0.106-1
NV_LIBNPP_VERSION=12.1.0.40-1
NCCL_VERSION=2.17.1-1
PWD=/workspace/SD_Next
DISABLE_AUTOLAUNCH=true
NV_CUDNN_PACKAGE=libcudnn8=8.9.0.131-1+cuda12.1
NVIDIA_DRIVER_CAPABILITIES=compute,display,graphics,utility,video
JUPYTER_SERVER_URL=http://localhost:8888/
NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
TZ=Europe/London
NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
NVIDIA_PRODUCT_NAME=CUDA
RUNPOD_CPU_COUNT=9
NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
LINES=77
NV_CUDA_CUDART_VERSION=12.1.105-1
HOME=/root
JUPYTER_PASSWORD=
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
COLUMNS=200
CUDA_VERSION=12.1.1
RUNPOD_POD_ID=pe3yrl6zuru4ws
NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
ENABLE_TENSORBOARD=1
RUNPODCTL_VERSION=v1.14.3
DISABLE_SYNC=true
NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
RUNPOD_MEM_GB=50
RUNPOD_PUBLIC_IP=69.30.85.120
NV_LIBNPP_DEV_VERSION=12.1.0.40-1
RUNPOD_VOLUME_ID=ksxds2pwbr
RUNPOD_GPU_COUNT=1
JUPYTER_SERVER_ROOT=/
TERM=xterm-256color
NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
RUNPOD_POD_HOSTNAME=pe3yrl6zuru4ws-64411155
venv_path=/workspace/venvs/stable-diffusion-webui
LIBRARY_PATH=/usr/local/cuda/lib64/stubs
NV_CUDNN_VERSION=8.9.0.131
RUNPOD_DC_ID=CA-MTL-1
SHLVL=2
PYXTERM_DIMENSIONS=80x25
NV_CUDA_LIB_VERSION=12.1.1-1
NVARCH=x86_64
NV_CUDNN_PACKAGE_DEV=libcudnn8-dev=8.9.0.131-1+cuda12.1
RUNPOD_GPU_NAME=NVIDIA+A40
NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
LC_CTYPE=C.UTF-8
REQUIRED_CUDA_VERSION=12.1
NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
RUNPOD_TCP_PORT_22=22149
NV_NVPROF_VERSION=12.1.105-1
RUNPOD_API_KEY=rpa_DPY5O5WLMGH9ZMFK4EF8HTVBFIMTKX5D96Q8G56O1fv5gz
PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NV_LIBNCCL_PACKAGE_NAME=libnccl2
NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1

VENV_PATH=/workspace/venvs/tennesseesdnext-lite
DEBIAN_FRONTEND=noninteractive

@vladmandic
Copy link
Owner

It's very long, so I'll have to split it in two.

fyi for future you can just drag and drop the log itself and it will be uploaded as a file

@vladmandic
Copy link
Owner

should be fixed in latest dev branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vladmandic @IowaSovereign