CI: "build Cosmo" step takes 20m and is repeated for all 10 groups

[franalbani]

Before submitting a PR with new software, one has to fork and test that github actions work. For some things, it is impossible to test locally and iterating against the CI is a must. This is painful because the long times it takes.

I have some questions before trying to improve it:

• do we need to use and build from scratch the latest cosmocc?
  • Can we just wget -qO $@ https://cosmo.zip/pub/cosmocc/cosmocc-3.9.2.zip?
• Can we use intermediate docker images for:
  1. ubuntu apt-get install stuff
  2. previous + cosmocc

Am I missing something?

Thanks!

[ahgamut]

I agree, there should be a focus on improving local testing.
Right now, I test on a local Linux machine/VM, and the only major thing to check for is the /zip requirement.

Perhaps we can have a separate github action for just PRs?

[jart]

SuperConfigure is what builds the cosmocc binaries. It also fetches an older version of cosmocc as part of its build process, to bootstrap building the cosmo repo.

If you're adding a project to a folder like editor/ or web/ then you shouldn't need to wait for GCC to compile from scratch to get feedback on your PR. @ahgamut what I would recommend doing is adding an additional GA runner for each folder, which only builds that folder. Then we just tell people, if you're adding an editor and the "editors" GA is green, then you're good to go.

[franalbani]

I still don't fully understand, but while I do, I want to ask:

1. Can this new Github action be as simple as cloning cosmopolitan and using tool/cosmocc or wget https://cosmo.zip/pub/cosmocc/cosmocc-3.9.2.zip and unzip?
2. Can superconfigure follow this (which looks simpler than the current CI)?
3. Is there a way to locally replicate the CI without:

• using a VM
• using docker
• requiring root access in the host system (e.g., for binfmt_misc)

I usually desire to make the CI as close to "local" as possible. I know how to do it with docker, but I also feel that philosophically goes against the spirit of cosmopolitan so I would love to learn how to do it without it.

[franalbani]

Also, when I try sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" inside an ubuntu-latest docker container I get sh: 1: cannot create /proc/sys/fs/binfmt_misc/register: Directory nonexistent.

Why is it possible in the Github Action?

[ahgamut]

I've added a Github Action that attempts builds based on the folders changed in the most recent commit, and creates a ZIP artifact with all the built binaries.

@franalbani the CI "build Cosmo" step now takes around 1m 30s, as seen here. Since the Action only builds binaries from folders that have changed, the build time is also reduced.

[ahgamut]

FWIW superconfigure was a proof-of-concept to show that gcc itself could be built with Cosmopolitan Libc on Github, and so the CI steps were set up around that.

My preference is to build cosmocc + most of Cosmopolitan Libc with the releases, just so we know that cosmocc is usable (and for testing experimental build features like jart/cosmopolitan#1344) . For commits and PRs, we have something a bit faster now.

[franalbani]

Many many thanks @ahgamut !

We are now iterating much faster.

what do you think about "freezing" the docker image?

[ahgamut]

I haven't used docker in a while. If freezing the image increases build speed and is not flaky, I'm happy to accept a PR.

[franalbani]

TIL: Github Actions do not use docker (at least as a default).

A runner is a server that runs your workflows when they're triggered. Each runner can run a single job at a time. GitHub provides Ubuntu Linux, Microsoft Windows, and macOS runners to run your workflows. Each workflow run executes in a fresh, newly-provisioned virtual machine.
Source.

That also explains:

Also, when I try sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" inside an ubuntu-latest docker container I get sh: 1: cannot create /proc/sys/fs/binfmt_misc/register: Directory nonexistent.

Why is it possible in the Github Action?

I will look into ways of achieving something similar (maybe with a cache action for apt), but I anticipate that Github may already be using internal caches for that.

The question now is: do we need to do apt-get update? This hits Ubuntu repositories and will frequently invalidate any cache.

[ahgamut]

Perhaps then we can have a docker image without apt-get update for PRs and commits, and use Github Actions as is for releases? Hmm, that doesn't seem much better than the present setup.

We're at 1m 30s now. Does freezing a docker image really give that much of a speed increase?

[franalbani]

The "docker way" is incompatible with the way in which we currently set binfmt_misc.

Looking at the last action run, the step support ape bins and SSL things took 33s and only 4 of those were downloading packages: Fetched 118 MB in 4s (32.1 MB/s). So caching apt seems pointless.

The step that takes 1m30 is build Cosmo (minimal). It is not a lot, but in a project we are working right now, it represents ~20% of total time.

That step may be cached, but first I have a question: why is the need to call make o//tool/build in .github/scripts/pr-cosmo. Isn't https://cosmo.zip/pub/cosmocc/cosmocc-4.0.2.zip ready out of the box?

[ahgamut]

why is the need to call make o//tool/build in .github/scripts/pr-cosmo. Isn't https://cosmo.zip/pub/cosmocc/cosmocc-4.0.2.zip ready out of the box?

Right now I'm trying to add a new tool to cosmocc for helping build this repo, so we're using o//tool/build. Generally, we have new versions of the tools used to build cosmocc in o//tool/build, so I always build the repo with that.

[franalbani]

Are you referring to renamestr?