Failed to set execute bit on remote files

[pbrezina]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I get this error message when using the podman connection plugin ansible_connection: podman

Failed to set execute bit on remote files (rc: 255, err: Error: open /var/lib/containers/storage/overlay-containers/3cd9e6f9a9c2ef61d3af4e63bee57571ea9f1d9b75782e56fabe2a26fade3566/userdata/d92176f8d558b6e60da40d8acf122d1ec9a7d21a0615e26aee40286cfa812485/exit/3cd9e6f9a9c2ef61d3af4e63bee57571ea9f1d9b75782e56fabe2a26fade3566: no such file or directory

Steps to reproduce the issue:

It does not happen everytime, it also happens on different places and different hosts so I don't have any reproducer. I never got this error locally (so far) but it happens in Github Actions on the Ubuntu runner: https://github.com/SSSD/sssd-ci-containers/runs/5496701829?check_suite_focus=true#step:4:585

Describe the results you received:

Error "Failed to set execute bit on remote files"

Describe the results you expected:

No error.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens only occasionally.

Version of the containers.podman collection:
Either git commit if installed from git: git show --summary
Or version from ansible-galaxy if installed from galaxy: ansible-galaxy collection list | grep containers.podman

Latest version installed via `sudo pip3 install ansible`

Output of ansible --version:

Latest version installed via `sudo pip3 install ansible`

Output of podman version:

Podman 3.4.2, ubuntu latest github runner

Output of podman info --debug:

N/A

Package info (e.g. output of rpm -q podman or apt list podman):

N/A

Playbook you run with ansible (e.g. content of playbook.yaml):

https://github.com/SSSD/sssd-ci-containers/blob/master/src/ansible/playbook_image_service.yml

Command line and output of ansible run with high verbosity

N/A

Please NOTE: if you submit a bug about idempotency, run the playbook with --diff option, like:

ansible-playbook -i inventory --diff -vv playbook.yml

N/A

Additional environment details (AWS, VirtualBox, physical, etc.):

Github Actions ubuntu latest runner.

[sshnaidm]

Seems like problem with mounting by podman. Is it possible to catch this with -vvvvv (and maybe even ANSIBLE_DEBUG=1) ?

[pbrezina]

I haven't been able to reproduce it so far with verbose mode enabled in 50 builds. Here is a successful log if it helps.

I will keep trying, but it looks to be a race condition which probability is lowered by the verbose mode.

[pbrezina]

@sshnaidm It can't be reproduced with ANSIBLE_DEBUG. Is there anything I can do to help?

The error must come from here and it seems to be only triggered when become: True is used. I remove become as a workaround since we don't really need it, but the issue should be fixed.

[sshnaidm]

@pbrezina can you please open an issue in Ansible and to put a link to this issue here? Seems like it's more Ansible thing, let's see what they say.

[pbrezina]

Hmm, it now happened again even without using become:

TASK [ssh_server : Configure SSH daemon] ***************************************
fatal: [samba]: FAILED! => 
  msg: |-
    Failed to set execute bit on remote files (rc: 255, err: Error: open /var/lib/containers/storage/overlay-containers/1a63c44f6ecc0e11b5118b845a4cb2f62c1d0074d66bf3b43810260dc61b54d5/userdata/e95a89bea4c85207c2214781d4dc03c92e1dcaa78823df9bc334a3609e275121/exit/1a63c44f6ecc0e11b5118b845a4cb2f62c1d0074d66bf3b43810260dc61b54d5: no such file or directory
    )
changed: [ipa]
changed: [ldap]
changed: [client]

[pbrezina]

I managed to get a failure with ANSIBLE_DEBUG=1, log is attached

failed.log

Lookup /var/lib/containers/storage/overlay-containers

[pbrezina]

Reply from ansible team:

Until proven it's a fault in Ansible we can't be investigating code outside of our codebase as we don't know how the podman collection works. Please follow it up in your other issue #389 before asking here.

[sshnaidm]

Yeah, as expected. I'll look at connection optimization in scope of #382, maybe will help.

[pbrezina]

I opened containers/podman#13613 since I think it might be an issue in podman itself.

[sshnaidm]

As I understand from the issue it should be solved by containers/podman@a1e2897 . I don't think we can do something, just to watch if it happens after a release. For today it's only in main branch, so need to wait I suppose.

[kobigi]

Hi @sshnaidm
i m having a similar issue.
this instability seems to came since we have started to use ansible with python 3
while we where using pyhton2.7 it was not shown - Maybe it will give a clue

BTW: the failures are random on any different ansible playbook , and not related to specific one
and worst then that is that if we resume the playbook it passes so very difficult to debug.

20:02:53 changed: [bvt-baremetal-allinone-1] => (item=lldpad.service)
20:02:53 fatal: [bvt-baremetal-allinone-0]: FAILED! => {"msg": "Failed to set execute bit on remote files (rc: 1, err: Couldn't open /dev/null: Operation not permitted\
20:02:53 )"}

[root@bvt-centralsite-centralsitemanager-0 cbis-admin (Active)]# ansible --version
ansible [core 2.13.10]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /bin/ansible
python version = 3.9.16 (main, May 31 2023, 12:21:58) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
jinja version = 3.1.2
libyaml = True

Any suggestions ?

[sshnaidm]

@kobigi I don't think I can help without additional details. Please open the new issue with all required info.