AzureMLException: Message: Failed to flush task queue within 300.0 seconds | dreambooth finetune text to image

[jyravi]

Operating System

Linux

Version Information

Running the sample in Compute Instance within the Azure ML workspace.
Running the sample provided in azureml-examples
pipeline component : diffusers_text_to_image_dreambooth_pipeline

Steps to reproduce

1. Ran the notebook as -is provided in azureml-examples "https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/finetune/text-to-image/diffusers-dreambooth-dog-text-to-image.ipynb"

Expected behavior

Finetune the model successfully.

Actual behavior

encountered error in the child job "text_to_image_dreambooth_finetune".

Encountered an internal ACFT error. Error Message/Code: AzureMLException:
Message: Failed to flush task queue within 300.0 seconds. Please set AZUREML_ARTIFACTS_DEFAULT_TIMEOUT environment variable to increase the timeout(in seconds)
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Failed to flush task queue within 300.0 seconds. Please set AZUREML_ARTIFACTS_DEFAULT_TIMEOUT environment variable to increase the timeout(in seconds) ",
"inner_error": {
"code": "ResourceExhausted",
"inner_error": {
"code": "Timeout"
}
}
}
}. Traceback: File "swallow_all_exceptions_decorator.py", line 68, in wrapper
return func(*args, **kwargs)
File "[Non-AutoML file]", line 1097, in [Non-AutoML function]
File "finetune_runner.py", line 252, in finetune_runner
mlflow.log_artifacts(component_args.output_dir, SettingParameters.DEFAULT_OUTPUT_DIR)
File "[Non-AutoML file]", line 1096, in [Non-AutoML function]
File "[Non-AutoML file]", line 1242, in [Non-AutoML function]
File "[Non-AutoML file]", line 570, in [Non-AutoML function]
File "[Non-AutoML file]", line 88, in [Non-AutoML function]
File "[Non-AutoML file]", line 97, in [Non-AutoML function]
File "[Non-AutoML file]", line 29, in [Non-AutoML function]
File "[Non-AutoML file]", line 55, in [Non-AutoML function]
File "[Non-AutoML file]", line 135, in [Non-AutoML function]
. Additional information: [Hidden as it may contain PII].
4.

Addition information

Talks about increasing the timeout. Not sure in compute instance how to find the file and change the default time out.