Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in train model #28

Open
yassirez-znafry opened this issue Feb 7, 2022 · 2 comments
Open

Error in train model #28

yassirez-znafry opened this issue Feb 7, 2022 · 2 comments

Comments

@yassirez-znafry
Copy link

yassirez-znafry commented Feb 7, 2022

I'm having trouble completing the getting_started example (getting_started.md) as the pipeline stops on the train (takes too long ≈ 60 min on train model job). Here are the last logs before canceling automatically (the file contains the entire logs:
Complete Logs.txt
):

2022-02-07T00:52:37.0050192Z WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/6b/b2/c0d62a3a91c13641e09af294c13fe16929f88dc5902718388cd9b292217f/azure_mgmt_authorization-0.52.0-py2.py3-none-any.whl
2022-02-07T00:52:37.0052090Z Downloading azure_mgmt_authorization-0.52.0-py2.py3-none-any.whl (112 kB)
2022-02-07T00:52:37.0052735Z
2022-02-07T00:57:40.9228879Z WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/a1/71/9a20913e92771b3c23564f1bea54d376d09fb30a75585087c70b769d75c8/azure_mgmt_authorization-0.51.1-py2.py3-none-any.whl
2022-02-07T00:58:41.5520782Z Downloading azure_mgmt_authorization-0.51.1-py2.py3-none-any.whl (111 kB)
2022-02-07T00:58:41.5521395Z
2022-02-07T00:59:42.2727333Z INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
2022-02-07T01:03:45.8869909Z Downloading azure_mgmt_authorization-0.51.0-py2.py3-none-any.whl (111 kB)
2022-02-07T01:09:52.4374279Z WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/6f/17/55b974603c16be89c7a7c16bac57b7bce48527bf1bebc3f116f7215176e6/azure_mgmt_authorization-0.50.0-py2.py3-none-any.whl
2022-02-07T01:09:52.4376241Z Downloading azure_mgmt_authorization-0.50.0-py2.py3-none-any.whl (81 kB)
2022-02-07T01:09:52.4376835Z
2022-02-07T01:26:07.6809069Z WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/67/e4/b3535daae30db9b3f73046a0c151c5c2ae2d2bff96ba0c28c1f26a21dbf1/azure_mgmt_authorization-0.40.0-py2.py3-none-any.whl
2022-02-07T01:26:07.6811091Z Downloading azure_mgmt_authorization-0.40.0-py2.py3-none-any.whl (38 kB)
2022-02-07T01:26:07.6811445Z
2022-02-07T01:39:04.9650251Z ##[error]The operation was canceled.
2022-02-07T01:39:04.9664245Z ##[section]Finishing: Train model

@ruqianq
Copy link

ruqianq commented Mar 21, 2022

@juliakm
Copy link
Contributor

juliakm commented Jun 28, 2022

I've updated the Train step to use pip 20.2.4 as a temporary fix - https://github.com/MicrosoftDocs/pipelines-azureml/blob/master/models/diabetes/config/train-conda.yml. The step still fails for me in the pipeline but it is successful in the Azure Machine Learning environment. I'm working on getting more help on this from the product team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants