incremental builds

[KiaraGrouwstra]

recently, garnix announced support for incremental builds, enabling caching also on the module level.
what would it take to support this in typhon?

[pnmadelaine]

Thank you for the very interesting read!

Short answer: Typhon is already able to support this workflow, it would only cost a little bit of noise on the repository and a lot of work tampering with actions. I would be happy to integrate relevant helpers to the library, although I can't make it a priority right now.

Long answer:

Typhon is designed around a set of very simple and predictable core components. Essentially, it evaluates Nix expressions, builds the produced derivations and runs the isolated (user-defined) actions afterwards. I am strongly opposed to add anything to this pattern that would not be idiomatic to any Nix-based CI system. In particular, incremental builds are a very interesting idea and arguably fundamental to bigger projects, but the implementation details might very well vary from project to project. Moreover, Garnix' solution will fundamentally obscure what is being evaluated, and worse, break reproducibility of the evaluations: currently, Typhon's evaluator is pure and only takes the locked jobset URL as an input. To accommodate incremental builds, it would need to either evaluate impurely or accept more inputs.

Following Typhon's principles, a workflow like this should be entirely defined at the user level, using actions. And I think that the action's hooks currently present in Typhon are already powerful enough to implement a very similar solution. Here is how I would personally approach this:

• Write your job's expressions and make them depend on a file called cache.nix in your repository. Either this file contains null and the expression should make the build non-incremental, or it contains a store path to the cache that can be used to make the build incremental. Also add a file called commit.txt.
• In your main branch and in your PR, keep commit.txt empty and cache.nix set to null.
• Edit the webhook action: for every PR, it should maintain another branch (let's call it the cache branch) by merging the PR into it after every commit and updating commit.txt to point to the corresponding commit on the PR. Then, create a new jobset for the cache branch instead of the original one.
• Add a post-build action that will update cache.nix on the cache branch to point to the output path of the build (only after successful builds of course).
• Finally, edit the status report action so that it uses commit.txt to report on the original commit instead of the commit on the cache branch.

To state the obvious: this is cumbersome to write and I don't except every one wanting to use the feature to do that kind of work and to deal with the (undoubtedly many) caveats of the above steps and the tweaks they require. Ideally, this kind of workflows would be implemented in Typhon's library and made available from a simple switch in a Nix configuration (edit: of course users would still have some work to do to write their expression in a way that can profit from the cache). Moreover, even though I would personally like to see progress in the "modules as packages" approach, I think incremental builds would be a very appealing feature for Typhon.

I think the above steps are a good starting place for future me (or anyone that would like to tackle such a PR :D) but I don't expect to have time to work on it in the near future.

[pnmadelaine]

It should also be noted that a lot of the bookkeeping I described could probably be mitigated if we implemented #39
But I still think this has the potential to create a lot more problems than it would solve.