Plain build of awx-ee fails

[Keith-Nowosielski-SJSU]

OS: RHEL 8.9
Ansible-builder version: 3.0.1
Python 3.9.18

Ansible version:
ansible 2.10.17
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/user1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.8 (default, Jan 5 2024, 08:58:17) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]

Ansible-builder version: 3.0.1
Python 3.9.18

Issue:

ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 179, in resolve
self.factory.preparer.prepare_linked_requirements_more(reqs)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 552, in prepare_linked_requirements_more
self._complete_partial_requirements(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 467, in _complete_partial_requirements
for link, (filepath, _) in batch_download:
File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/download.py", line 183, in call
for chunk in chunks:
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/progress_bars.py", line 52, in _rich_progress_bar
with progress:
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/progress.py", line 1169, in enter
self.start()
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/progress.py", line 1160, in start
self.live.start(refresh=True)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/live.py", line 132, in start
self._refresh_thread.start()
File "/usr/lib64/python3.9/threading.py", line 899, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
Error: error building at STEP "RUN $PYCMD -m pip install --no-cache-dir bindep pyyaml requirements-parser": error while running runtime: exit status 2

### Steps to reproduce:

git clone https://github.com/ansible/awx-ee.git

cd awx-ee

ansible-builder build -v3 -t quay.io/ansible/awx-ee ####Same instructions as in the readme.txt on this repo.

### FULL OUTPUT:

(ansible-builder) [user1@server-01 awx-ee]$ ansible-builder build -v3 -t quay.io/ansible/awx-ee
Ansible Builder is generating your execution environment build context.
File context/_build/requirements.yml is already up-to-date.
File context/_build/requirements.txt is already up-to-date.
File context/_build/bindep.txt is already up-to-date.
File context/_build/scripts/assemble is already up-to-date.
File context/_build/scripts/install-from-bindep is already up-to-date.
File context/_build/scripts/introspect.py is already up-to-date.
File context/_build/scripts/check_galaxy is already up-to-date.
File context/_build/scripts/check_ansible is already up-to-date.
File context/_build/scripts/entrypoint is already up-to-date.
Ansible Builder is building your execution environment image. Tags: quay.io/ansible/awx-ee
Running command:
podman build -f context/Containerfile -t quay.io/ansible/awx-ee context
STEP 1: FROM quay.io/centos/centos:stream9 AS base
STEP 2: USER root
--> Using cache 0c4ede4117d7a3fff5b824d82d5d0db6429a85eb7ec0e861a097329580c9b282
STEP 3: ARG EE_BASE_IMAGE
--> Using cache 233b790c158abf6bde94e2b94226e798e014334a7ccd5a557b45e3da1df6268c
STEP 4: ARG PYCMD
--> Using cache 3b595db6c20c94a2904fdedcc271713df9f94a006a90966c1a1e4a060e9c240d
STEP 5: ARG PKGMGR_PRESERVE_CACHE
--> Using cache 33020a8ebfbcd186d5cf55f6b596ba19e9326b058caae2fb1909d6897c40803a
STEP 6: ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS
--> Using cache cc239a2a18a2ee913c4b982e67ef65bd56da55e0d741c48038f4265f7aeb32dc
STEP 7: ARG ANSIBLE_GALAXY_CLI_ROLE_OPTS
--> Using cache b021e216b345584f901c1b7e7f35dabda075a6022a082e90eaf7fab52b181643
STEP 8: ARG ANSIBLE_INSTALL_REFS
--> Using cache aca95036113a33b2f0d493fc141a63b77a3877da0b0935428aec4417232a1f25
STEP 9: ARG PKGMGR
--> Using cache bb9d2e1eb21b4329f61c63e7ba96cefb2cb167f9ca1583cbffd66018557c5bff
STEP 10: RUN $PYCMD -m ensurepip
--> Using cache acbeadb8b71390fd47a6a326d5d71c6eb47de2c50487fc905f56ed053494dbc4
STEP 11: RUN $PYCMD -m pip install --no-cache-dir $ANSIBLE_INSTALL_REFS
--> Using cache 0f8361bfffb0b14b18d15601ee24f842d54757ad852762307b61c94440611b87
STEP 12: COPY _build/scripts/ /output/scripts/
--> Using cache e578b34bf09ead4689e75d577671774d3960e6cd0d97bd1e3484f41f1461817f
STEP 13: COPY _build/scripts/entrypoint /opt/builder/bin/entrypoint
--> Using cache 70037413fb122f7c490a9285fa8307c33c2ffcedba6e2cfc94cc10265defac6d
STEP 14: RUN $PYCMD -m pip install -U pip
--> Using cache 00ce6c82dcc74959d0b4b22b6757879bedf27c28b583e21808c2a8c5f62d522c
STEP 15: FROM 00ce6c82dcc74959d0b4b22b6757879bedf27c28b583e21808c2a8c5f62d522c AS galaxy
STEP 16: ARG EE_BASE_IMAGE
--> Using cache 9e1b289dfb1e1008d7602884ff6bcf621c3172769a138455e0c9d994e612adad
STEP 17: ARG PYCMD
--> Using cache 0b03f40ca8284bca2b6d1e21b9591ec7bbbb72624e68968652b9199fc6f7f291
STEP 18: ARG PKGMGR_PRESERVE_CACHE
--> Using cache 68492f2500c30048c656d76ef101da157f612d952ca8ea9a39c3014bc3226658
STEP 19: ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS
--> Using cache b67664b4ab51fdbfca2963723145ed82d71d989ed1951ee9c108df51e0062d23
STEP 20: ARG ANSIBLE_GALAXY_CLI_ROLE_OPTS
--> Using cache 32fba909414485ac01f2e2851400d4f6503e01acf905b13431a6f4c5d0364448
STEP 21: ARG ANSIBLE_INSTALL_REFS
--> Using cache f0400c71e245efaf73c35db2b4b2dd714013c2f0b145191667e7df6c2d0b23e0
STEP 22: ARG PKGMGR
--> Using cache 9726aa26c6c279fc854a962c7fa5dfa82d8aa62ed6b03f6e3d7dfa04990cbc7d
STEP 23: RUN /output/scripts/check_galaxy
--> Using cache 9885acc04d6edfed872bd5deb1c407373a1dd5ac6039037c88229c8078a96355
STEP 24: COPY _build /build
--> Using cache f1d6b9476e87c3f3d816c5821bf2bf6a6f26744450d41ee649fe37aca56c9810
STEP 25: WORKDIR /build
--> Using cache 4ac460491b80322198eb473ae386aeea3824ca7f1a135c015e4b07cf2d21b7d6
STEP 26: RUN ansible-galaxy role install $ANSIBLE_GALAXY_CLI_ROLE_OPTS -r requirements.yml --roles-path "/usr/share/ansible/roles"
--> Using cache 1ed9139afe54030448f47e503c229a632d3ad149fcdc839176427794d3bdbaa3
STEP 27: RUN ANSIBLE_GALAXY_DISABLE_GPG_VERIFY=1 ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path "/usr/share/ansible/collections"
--> Using cache feb181f1dad5c6e6b209165b6750e00af9532c38dc8433ef193c4ef8c5586c8a
STEP 28: FROM 00ce6c82dcc74959d0b4b22b6757879bedf27c28b583e21808c2a8c5f62d522c AS builder
STEP 29: WORKDIR /build
--> Using cache f44b3d9d21434af493f885956e6f774fbf8083e0788ced68f295b859c497c874
STEP 30: ARG EE_BASE_IMAGE
--> Using cache 516073413372ac04d04b4c29f9cfc2a19c09d496a99f405ee5bc1ed1befe63f1
STEP 31: ARG PYCMD
--> Using cache b4c113723d4fe1e99fe0f80bff66ed37500b702055f291c7f909552bc3751307
STEP 32: ARG PKGMGR_PRESERVE_CACHE
--> Using cache 229c8286b2d42e7609bb07f263021045b18867c10b47cba743a5c572cff4b566
STEP 33: ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS
--> Using cache 8454c3d537cf867276bd86ab59a326e57cfe6a1bd47c8cb3958453a09cea0c90
STEP 34: ARG ANSIBLE_GALAXY_CLI_ROLE_OPTS
--> Using cache e9df7bda46d1214ba4ec7aea4cfa04267d1e03fca33ea4658397c2e53ed11c6f
STEP 35: ARG ANSIBLE_INSTALL_REFS
--> Using cache 6a1b7a3d64d95e7882263dbb5e03c8f5aff7af5c09761ff1e97541b6925d4b8c
STEP 36: ARG PKGMGR
--> Using cache 1dfec9914414584be0f23a849dccf94d1d51e85da8150131424778ad4329de4e
STEP 37: RUN $PYCMD -m pip install --no-cache-dir bindep pyyaml requirements-parser
Collecting bindep
Downloading bindep-2.11.0-py2.py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: pyyaml in /usr/local/lib64/python3.9/site-packages (6.0.1)
Collecting requirements-parser
Downloading requirements_parser-0.5.0-py3-none-any.whl.metadata (3.1 kB)
Collecting Parsley (from bindep)
Downloading Parsley-1.3-py2.py3-none-any.whl.metadata (4.1 kB)
Collecting pbr>=2.0.0 (from bindep)
Downloading pbr-6.0.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting distro>=1.7.0 (from bindep)
Downloading distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
Requirement already satisfied: packaging in /usr/local/lib/python3.9/site-packages (from bindep) (23.2)
Collecting types-setuptools>=57.0.0 (from requirements-parser)
Downloading types_setuptools-69.1.0.20240302-py3-none-any.whl.metadata (1.6 kB)
Downloading bindep-2.11.0-py2.py3-none-any.whl (34 kB)
Downloading requirements_parser-0.5.0-py3-none-any.whl (18 kB)
Downloading distro-1.9.0-py3-none-any.whl (20 kB)
Downloading pbr-6.0.0-py2.py3-none-any.whl (107 kB)
ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 179, in resolve
self.factory.preparer.prepare_linked_requirements_more(reqs)
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 552, in prepare_linked_requirements_more
self._complete_partial_requirements(
File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 467, in _complete_partial_requirements
for link, (filepath, _) in batch_download:
File "/usr/local/lib/python3.9/site-packages/pip/_internal/network/download.py", line 183, in call
for chunk in chunks:
File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/progress_bars.py", line 52, in _rich_progress_bar
with progress:
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/progress.py", line 1169, in enter
self.start()
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/progress.py", line 1160, in start
self.live.start(refresh=True)
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/rich/live.py", line 132, in start
self._refresh_thread.start()
File "/usr/lib64/python3.9/threading.py", line 899, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
Error: error building at STEP "RUN $PYCMD -m pip install --no-cache-dir bindep pyyaml requirements-parser": error while running runtime: exit status 2

[Keith-Nowosielski-SJSU]

It's not a lack of resources I had nearly 6GB of memory free initially and I doubled my memory and CPU cores and still get the same error.

[Keith-Nowosielski-SJSU]

Upgraded podman from 1.6.x to 4.6.1 which solved the problem.