[BUG] Docker validation failed due to latest docker-py update

[peterzhuamazon]

This is a followup to the issue #4929.

The failures:
https://build.ci.opensearch.org/job/distribution-validation/974/console

+ ./validation.sh --version 2.16.0 --distribution docker --arch x64 --projects opensearch opensearch-dashboards --docker-source dockerhub --using-staging-artifact-only
Installing dependencies in . ...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (6b098c)...
Running ./src/run_validation.py --version 2.16.0 --distribution docker --arch x64 --projects opensearch opensearch-dashboards --docker-source dockerhub --using-staging-artifact-only ...
2024-08-09 20:55:00 INFO     Docker daemon is running
2024-08-09 20:55:00 INFO     Image does not exist at local, proceed with Pull from the dockerhub
2024-08-09 20:55:00 INFO     Pulling opensearchstaging/opensearch:2.16.0 from the dockerhub
2024-08-09 20:55:24 INFO     Image is pulled at local : sha256:a6cc0d11683dd224aee4179ceb8958f0536e2d2f99f3862f8e8fd150fb1aed7b
 : opensearchstaging/opensearch : 2.16.0
2024-08-09 20:55:24 INFO     Image does not exist at local, proceed with Pull from the dockerhub
2024-08-09 20:55:24 INFO     Pulling opensearchstaging/opensearch-dashboards:2.16.0 from the dockerhub
2024-08-09 20:55:49 INFO     Image is pulled at local : sha256:5775918002fa886f7ddaef811cb70dc4d9725de60ac8f457a21a977c38f5d8a3
 : opensearchstaging/opensearch-dashboards : 2.16.0
2024-08-09 20:55:49 INFO     Docker images are downloaded
2024-08-09 20:55:49 INFO     Changing "opensearchproject/opensearch:2" to "sha256:a6cc0d11683dd224aee4179ceb8958f0536e2d2f99f3862f8e8fd150fb1aed7b" in /tmp/tmpphc1gjur/docker-compose.yml
2024-08-09 20:55:49 INFO     Changing "opensearchproject/opensearch-dashboards:2" to "sha256:5775918002fa886f7ddaef811cb70dc4d9725de60ac8f457a21a977c38f5d8a3" in /tmp/tmpphc1gjur/docker-compose.yml
2024-08-09 20:55:50 INFO     Removing /tmp/tmpphc1gjur
Traceback (most recent call last):
  File "/var/jenkins/workspace/distribution-validation/src/validation_workflow/validation.py", line 59, in run
    return self.download_artifacts() and self.installation() and self.start_cluster() and self.validation() and self.cleanup()
  File "/var/jenkins/workspace/distribution-validation/src/validation_workflow/docker/validation_docker.py", line 94, in validation
    raise Exception('The container failed to start. Exiting the validation.')
Exception: The container failed to start. Exiting the validation.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/jenkins/workspace/distribution-validation/./src/run_validation.py", line 38, in <module>
    sys.exit(main())
  File "/var/jenkins/workspace/distribution-validation/./src/run_validation.py", line 29, in main
    test_result = ValidationTestRunner.dispatch(docker_args, args.distribution, work_dir).run()
  File "/var/jenkins/workspace/distribution-validation/src/validation_workflow/validation.py", line 61, in run
    raise Exception(f'An error occurred while running the validation tests: {str(e)}')
Exception: An error occurred while running the validation tests: The container failed to start. Exiting the validation.

Thanks.

[peterzhuamazon]

After a while I finally able to reproduce the issue and get the actual failure msg:

$ docker-compose -f docker-compose.yml up
Traceback (most recent call last):
  File "/home/ec2-user/.local/bin/docker-compose", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/home/ec2-user/.local/lib/python3.9/site-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/compose/cli/docker_client.py", line 124, in docker_client
    kwargs = kwargs_from_env(environment=environment, ssl_version=tls_version)
TypeError: kwargs_from_env() got an unexpected keyword argument 'ssl_version'

[peterzhuamazon]

Seems like due to this issue:

• https://github.com/docker/docker-py/releases/tag/7.0.0

More:

• Ansible / docker==7.0.0 docker compose up -> kwargs_from_env ssl_version docker/docker-py#3194 (comment)

[peterzhuamazon]

Our AL2023 runner havent been updated for a while, so the latest update pulled in the latest docker-py changes. Thus shows error.

[peterzhuamazon]

Might need to force back to docker 6.1.3 version on PyPI.

[peterzhuamazon]

Still failed:

Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/ec2-user/.local/lib/python3.9/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/ec2-user/.local/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/requests/adapters.py", line 534, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

[peterzhuamazon]

Issues here:

• Breaks with requests 2.32.0: Not supported URL scheme http+docker docker/docker-py#3256

After switching back to the requests==2.28.1, it runs.

[peterzhuamazon]

The related bug fix in docker-py force to use 7.1.0 at least:

• Fix for requests 2.32 docker/docker-py#3257

So we have to pin both docker in PyPI as well as request altogether.

[peterzhuamazon]

Runs now:

2024-08-13 19:58:38 INFO     Checking if cluster is ready for API test in every 5 seconds


2024-08-13 19:58:38 INFO     Sleeping 5sec for retry 1/20
2024-08-13 19:58:43 ERROR    Error connecting to opensearch cluster API (https://localhost:9200): HTTPSConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1143)')))

......

2024-08-13 19:58:59 INFO     All tests Pass : There are 4/4 test cases Pass
2024-08-13 19:59:02 INFO     cleanup is complete
2024-08-13 19:59:02 INFO     Docker validation cleanup is complete
2024-08-13 19:59:02 INFO     final test_result = True

[peterzhuamazon]

Test run fixed:
https://build.ci.opensearch.org/job/distribution-validation/976/console