Fix failing CI tests

[apyrgio]

Our CI tests have recently started to fail on their own, due to upstream changes. See an example CI run here.

We see 3 tests that are failing:

1. run-lint: The poetry install command fails with exit code 1, and no log message.

2. build-debian-bookworm: The "Install dependencies (deb)" task fails with Unable to locate package python-all.

3. convert-test-docs: The "Install poetry dependencies" step fails with:

   ERROR: launchpadlib 1.10.13 requires testresources, which is not installed.
   ERROR: importlib-resources 5.10.1 has requirement zipp>=3.1.0; python_version < "3.10", but you'll have zipp 1.0.0 which is incompatible.
   

[apyrgio]

Fixing build-debian-bookworm

Problem

The python-all package refers to Python2 versions, which we don't support. Since Debian Bookworm is relatively bleeding edge, and Python 2 is EOL, it has completely dropped Python 2 packages, and therefore this package as well.

If we remove the python-all package from the list of the packages we install, then the Dangerzone package builds successfully in Debian Bookworm. However, it fails to build on Ubuntu Focal, because it somehow requires python-all. This may have to do with an stdeb issue from 2020: astraw/stdeb#153. This has been fixed in version 0.9.1, but Ubuntu Focal provides 0.8.5.

Solution

We can overcome this issue by installing python-all solely for Ubuntu Focal. We can add this as a separate step only for the build-ubuntu-focal task.

While working on this issue, it became obvious that we can do something about the extra packages we have to install in this step, and most probably are not required. Read the following section for more details.

Simplifying build-debian-bookworm

What we did is that we used a vanilla debian:bookworm container, mounted Dangerzone there, and attempted to run our ./install/linux/build-deb.py script there. Through trial and error, we found that the necessary packages for building the Dangerzone .deb are python3, python3-setuptools, python3-stdeb, and dh-python.

The rest of the packages are necessary if we run the unit tests with dh_auto_test, which is the default behavior. This behavior can be disabled though with the DEB_BUILD_OPTIONS=nocheck envvar. See also:

• Skip tests astraw/stdeb#89
• How do we disable dh_auto_test from stdeb.cfg so that tests dont run on Debian build servers? astraw/stdeb#178

By disabling tests while building the package, we don't sacrifice much, since unittest never actually ran the Dangerzone tests, as they are PyTest test cases.

[apyrgio]

Fixing run-lint

I tried to reproduce the error locally, in a debian:bullseye container. Unfortunately, I didn't manage to, since the poetry install command always ran to completion. I tested if the issue manifests when the standard streams are not TTYs, and the command still succeeded.

Possible causes for this issue may be:

1. Changes to the CircleCI runners
2. Changes to the Poetry package on PyPI
   • Update: This is indeed the root cause of this error: Poetry 1.3 dies when run with a TTY reporting size 0/0 python-poetry/poetry#7184

In both cases, we would need to continue the debugging in a CircleCI runner. There is a way to avoid all this hassle though, and even improve this step:

1. Change the base image for this step from debian:bullseye to debian:bookworm.
2. Install python3-poetry from the official repos, instead of downloading it from PyPI. This has the benefit of using more stable versions of Poetry than the latest ones in PyPI.
   • By default, installing python3-poetry brings lots of packages with it, some of them GUI related. We can skip those with --no-install-recommends.
3. Because Debian Bookworm provides python-poetry==1.2.2, we can use the --only dev flag, and install only the dev dependencies.

[apyrgio]

Fixing convert-test-docs

Turns out that this step does not fail with this error:

ERROR: launchpadlib 1.10.13 requires testresources, which is not installed.
ERROR: importlib-resources 5.10.1 has requirement zipp>=3.1.0; python_version < "3.10", but you'll have zipp 1.0.0 which is incompatible.

Instead, it fails for the same reason that run-lint fails, i.e., due to a Poetry 1.3 bug: python-poetry/poetry#7184. A simple workaround for now is to pin Poetry to 1.2.2, and then remove the pin once it's fixed upstream, as it affects only installations from PyPI.

Note that we also tested replacing machine runner ubuntu-2004 with ubuntu-2204, and install Poetry from the official repos. Turns out that they provide only Poetry 1.1, but we need Poetry 1.2, which supports the group syntax.

[deeplow]

Fixing debian-bookworm

With the above changes, we greatly simplify the build dependencies, and shave off roughly half a minute from each build step.

Thanks for the careful analysis. I am inclined to agree with this assessment. Minimizing the dependencies will also decrease the chance of dependency issues in the future.

Fixing run-lint

In both cases, we would need to continue the debugging in a CircleCI runner. There is a way to avoid all this hassle though, and even improve this step:

Great! Since this is only linting, it shouldn't matter that much which version we're running.