[RFC] meson2hermetic in mesonbuild repo

[gurchetansingh]

We are proposing porting the meson2hermetic tool to the mesonbuild repository.

It turns the meson build specification into hermetic build rules.

Where will we use this tool?
The use case will be initially Mesa3D integration in Fuchsia/Android, but we want to support a few other projects (QEMU, gfxstream_backend) as well.

The first two hermetic build systems we want to support are Soong and Bazel. The other major ones are buck2 and GN.

Hermetic Build Systems: why not just use Meson + NinjaBuild?

Hermetic build systems tend to target a different use case (very large projects) than meson. The numbers are irresistible: all of Android can be built in ~30 minutes using the tricks allowed by hermeticization.

The issue is these systems need a lot of resources to set-up and are not user friendly. Meson itself has benefits: a terrific API, specification, great for distros + developers. We want a bridge between these two worlds. This will allow meson.build files to be the source of truth for the various hermetic build systems. For example, llvm-project supports both GN and Bazel.

What if we can add meson to llvm-project (they allow alternate build systems in the "peripheral” support tier), and then use meson2hermetic to convert to Bazel/GN? Maybe that could show the strengths of meson versus CMake in that project? That’s the sort of thing we can do with this tool, if it works well and consistently.

Current approach: It kind of works, but we can do better

Implementation-wise, our script is a two-step process:

1. Use a meson2python converter, since the meson API is mostly python-like:

Overload the meson API in the python script with our custom functions. For example, see the “shared_library” overload.

3. Run the output python script multiple times
   (a) The meson arguments, toolchain (compiler, host machine), dependencies are given by a .toml file
   (b) Save state in from each run
   (c) After all runs are complete, construct hermetic build rules

We did it this way since we just needed something quick and dirty, and it works well enough to unblock some work. We feel we can greatly improve the script by leveraging the internal meson APIs available in the meson repository. We want human-readable output from the tool, as if a knowledgeable human had hand-written the hermetic build tools.

Improvement Idea #1: Use Interpreter class from tool directly
A keen observer can see (1) can be removed almost certainly. It seems we can overload the meson API in the AST code.

Improvement Idea #2: Write another Backend
There is also the possibility of writing another backend. For example, a “Bazel” backend was added.

That approach uses a shadow ninja build (which is avoidable), and it doesn’t do the multiple invocations of our current approach. But we think a “hermetic” meson backend is possible, if there is a way to invoke a backend multiple times with different arguments to construct the full build rules. The APIs used by meson introspection also look interesting.

Improvement Idea #3: Somehow access the Build class

We need the state (static libraries, include dirs, custom targets, paths in $SRC_DIR) from multiple builds to create the hermetic build rules. Use Build class?

Dependencies
We use jinja2 for templates, ruff for the Linter, and tomlib for configuration. We'll transition to internal meson templates and meson uses pylint, but tomlib is core in python 3.11. We envisage this as an additional helper program:

https://mesonbuild.com/Contributing.html#external-dependencies

so we’ll probably keep that around.

Expected kLOC
2kLoC -> 5kLoC, the less the better.

Proposed intermediate state
Like cmake2meson.py, our hunch is that meson2hermetic.py should live in the {$MESON_SRC}/tools directory. It needs to access APIs from the ${MESON_SRC}/mesonbuild directory though.

Ideal end state
Maybe after the tool has reached sufficient maturity, we can make it an official meson API.

meson convert [mesa-aosp.toml, mesa-fuchsia.toml, qemu.toml] ${SRC_DIR} ${TARGET_DIR}

HELP NEEDED FOR DESIGN: meson internals
We don’t understand the meson codebase well enough to know the best approach, but feel it’s doable given Meson’s current abstractions.

We would like your advice on how an experienced Meson developer would implement the tool. We can add the implementation, tests, make sure the tool is actively used etc. – but we need your feedback on the design.

[gurchetansingh]

CC: @eli-schwartz @dcbaker @bonzini for insights/opinions. There was another discussion thread, but Github deleted the user that created it erroneously, apologies for any confusion.

[dcbaker]

Just a few initial thoughts:

We use jinja2 for templates, ruff for the Linter, and tomlib for configuration. We'll transition to internal meson templates and meson uses pylint, but tomlib is core in python 3.11. We envisage this as an additional helper program:

We use toml in our rust/cargo converter, and we do a dance of "try tomllib, if that fails try tomli, if that fails look for a binary called toml2json, convert the toml to json and load that". If that fails, we just give up. Such an approach might be possible as well. we could potentially move that bit of code out of the cargo code into our common lib.

We also use flake8, which ruff is more-or-less attempting compatibility with for it's linter. I've used ruff a few times, but I don't really know what the delta between ruff and flake8 is ATM. For me the biggest linting concern is to use mypy or pyright, as we've found that catches a lot of corner cases.

I've wished for jinja in our meson init code. Maybe we can consider allowing jinja for secondary features?

Improvement Idea #1: Use Interpreter class from tool directly
A keen observer can see (1) can be removed almost certainly. It seems we can overload the meson API in the AST code.

One thing to be aware of: There's a fair amount of code in the Interpreter to handle deprecated and buggy DSL constructs. Manipulating the AST directly means that you'll need to potentially handle those issues yourself.

Improvement Idea #2: Write another Backend

Back when Android started migrating to Soong from android.mk I had inquired and half offered to write a Soong backend for Meson, specifically for Mesa. I'm not opposed to this idea necessarily, as long as there' someone willing to own that backend for bug fixes and review purposes.

Improvement Idea #3: Somehow access the Build class

I'd really prefer to keep this an implementation detail. There's a lot of really messy code in build.py especially that I've been trying to clean up for years. Actually, I just sent a PR to clean up more of it.

What if we can add meson to llvm-project (they allow alternate build systems in the "peripheral” support tier), and then use meson2hermetic to convert to Bazel/GN? Maybe that could show the strengths of meson versus CMake in that project? That’s the sort of thing we can do with this tool, if it works well and consistently.

I could probably be convinced to write those patches.

[gurchetansingh]

I'm not opposed to this idea necessarily, as long as there' someone willing to own that backend for bug fixes and review purposes.

Terrific, we do have a rotation setup for maintenance of Android graphics in particular, maintenance of the backend could be one the tasks.

We'll investigate the extra "hermetic" backend approach further then.

I could probably be convinced to write those patches.

The parts of LLVM we are most interested in are the ones required for building Lavapipe and RADV. If LLVM had a Meson backend, I can argue for a LLVM + Mesa build in ci.android.com hourly builders (no guarantee we get it).

[dcbaker]

I suspect you also already need, or will need, Clang support. We (Intel) and the Apple drivers are both using GPU based code already as a build time dependency.

Here's a WIP branch: https://github.com/dcbaker/llvm-project/tree/wip/2025-01/meson-build-system

[gurchetansingh]

Nice!

The most obvious target is to transition the Mesa CI to use a Meson llvm-project build instead of CMake, and also pro-Meson distros (pretty much all of them?).

I also noticed the Android toolchain is built using CMake:

https://android.googlesource.com/toolchain/llvm_android/+/refs/heads/main/src/llvm_android/configs.py#131

in some sort of container, rather than the normal Android build. Essentially, if Bazel (generated by meson2hermetic) is indeed faster than CMake + Ninja, that could be a path forward for that use case.

[dcbaker]

I've squashed everything together, since I've gotten a ton of the simple "write code" work done. There's still more to be done, but I really want to be able to test this in Mesa, and that's going to require some Meson changes, so I'm probably going to pivot to that for a bit.

[eli-schwartz]

Improvement Idea #2: Write another Backend There is also the possibility of writing another backend. For example, a “Bazel” backend was added.

That approach uses a shadow ninja build (which is avoidable), and it doesn’t do the multiple invocations of our current approach. But we think a “hermetic” meson backend is possible, if there is a way to invoke a backend multiple times with different arguments to construct the full build rules. The APIs used by meson introspection also look interesting.

Possibly of interest, the "genvslite" backend is essentially the Visual Studio backend except it produces multiple build directories (debug and debugoptimized and release) and then merges them together via a coordinating ninja. It is similar to cmake's multi-config generator, if I understand correctly. The idea is to have Visual Studio integration for all buildtypes directly inside the Visual Studio GUI...

The trick it uses is to repeatedly run the main generate() routine, passing an extra context parameter that accumulates values across each run. It is a bit of a special case but perhaps Soong/Bazel are similar types of special case?

[gurchetansingh]

Yes, that sounds very much like what we are looking for! We'll take a deeper dive and have a clearer RFC in coming weeks, if the "backend" approach is the best one. It's just a matter of a invoking a backend multiple times from scripts in tools directory, I feel.

[bonzini]

As an aside, I sometimes wondered what it would look like if Meson used Starlark as its DSL... :)