ansible_connection=docker and async poor performance

[lucab85]

SUMMARY

Hello,

When running the following playbook against a node with ansible_connection=local or ansible_connection=ssh, the async works as expected and perform well.

When running against a container, using ansible_connection=docker or ansible_connection=podman it took more time.

The ansible Engineer Team excluded any incompatibility between async and docker connector plugin with pipeline enable on #77046

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible_connection=docker

ANSIBLE VERSION

ansible --version
ansible 2.9.27
  config file = /opt/awxrpm/sclbuilder-podman/ansible.cfg
  configured module search path = ['/opt/awxrpm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Mar 18 2021, 08:58:41) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]

COLLECTION VERSION

Installing 'community.general:4.4.0' to '/root/.ansible/collections/ansible_collections/community/general'

CONFIGURATION

EMPTY

OS / ENVIRONMENT

RHEL 8

STEPS TO REPRODUCE

---
- hosts: example
  vars:
    durations:
      - 1
      - 2
      - 3
      - 4
      - 5
  tasks:
  - name: Async sleeping for batched_items
    ansible.builtin.command: sleep {{ async_item }}
    async: 45
    poll: 0
    loop: "{{ durations }}"
    loop_control:
      loop_var: "async_item"
    register: async_results

  - name: Check sync status
    async_status:
      jid: "{{ async_result_item.ansible_job_id }}"
    loop: "{{ async_results.results }}"
    loop_control:
      loop_var: "async_result_item"
    register: async_poll_results
    until: async_poll_results.finished
    retries: 30

EXPECTED RESULTS

Comparable performance on ansible_connection=local and ansible_connection=docker

ACTUAL RESULTS

• ansible_connection: local

$ time ansible-playbook test.yml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] ******************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Async sleeping for batched_items] *******************************************************************************************************************************************************************************************************************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
changed: [localhost] => (item=4)
changed: [localhost] => (item=5)

TASK [Check sync status] **********************************************************************************************************************************************************************************************************************************************
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '796033445730.3505116', 'results_file': '/opt/awxrpm/.ansible_async/796033445730.3505116', 'changed': True, 'failed': False, 'async_item': 1, 'ansible_loop_var': 'async_item'})
FAILED - RETRYING: Check sync status (30 retries left).
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '99334922661.3505135', 'results_file': '/opt/awxrpm/.ansible_async/99334922661.3505135', 'changed': True, 'failed': False, 'async_item': 2, 'ansible_loop_var': 'async_item'})
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '498307812124.3505155', 'results_file': '/opt/awxrpm/.ansible_async/498307812124.3505155', 'changed': True, 'failed': False, 'async_item': 3, 'ansible_loop_var': 'async_item'})
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '899582700725.3505175', 'results_file': '/opt/awxrpm/.ansible_async/899582700725.3505175', 'changed': True, 'failed': False, 'async_item': 4, 'ansible_loop_var': 'async_item'})
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '810355979454.3505195', 'results_file': '/opt/awxrpm/.ansible_async/810355979454.3505195', 'changed': True, 'failed': False, 'async_item': 5, 'ansible_loop_var': 'async_item'})

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


real    0m10.455s
user    0m4.929s
sys     0m1.227s

• ansible_connection: docker

$ time ansible-playbook -i ../container-dynamic test.yml 

PLAY [example] *****************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host example is using the discovered Python interpreter at /usr/bin/python3.6, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [example]

TASK [Async sleeping for batched_items] *******************************************************************************************************************************************************************************************************************************
changed: [example] => (item=1)
changed: [example] => (item=2)
changed: [example] => (item=3)
changed: [example] => (item=4)
changed: [example] => (item=5)

TASK [Check sync status] **********************************************************************************************************************************************************************************************************************************************
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '30977658294.7303', 'results_file': '/root/.ansible_async/30977658294.7303', 'changed': True, 'failed': False, 'async_item': 1, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '9649964613.7342', 'results_file': '/root/.ansible_async/9649964613.7342', 'changed': True, 'failed': False, 'async_item': 2, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '286584213193.7383', 'results_file': '/root/.ansible_async/286584213193.7383', 'changed': True, 'failed': False, 'async_item': 3, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '952408450077.7421', 'results_file': '/root/.ansible_async/952408450077.7421', 'changed': True, 'failed': False, 'async_item': 4, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '50375873698.7463', 'results_file': '/root/.ansible_async/50375873698.7463', 'changed': True, 'failed': False, 'async_item': 5, 'ansible_loop_var': 'async_item'})

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************************
example     : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


real    0m30.483s
user    0m14.705s
sys     0m8.448s

[felixfontein]

I'm not sure how this should be a bug. The connection plugin is working fine. Executing commands in docker containers is a lot slower than executing commands locally, that's why it takes longer.

[felixfontein]

(Also this collection does not contain the podman connection plugin, you need to check out the community.podman collection for that.)

[felixfontein]

@lucab85 could you please elaborate why exactly this is a bug? If you do not provide more information, I will close this.

[lucab85]

Could we improve the connection performance? I know that performance are influence by so many factors and probably you need to focus also on maintaining compatibility between different version of Docker.
@felixfontein I appreciate your message.
I already open a similar bug report for ansible_connection=podman containers/ansible-podman-collections#382

[felixfontein]

@lucab85 but how should we improve performace? Can you point out a possible optimization, or way to improve performance? (Also this would be a feature request, not a bug report, since I don't see a bug here.)