Bazel Remote Output Service

[coeuvre]

We propose to add a new flag --experimental_remote_output_service which accepts a URI pointing to a remote output service server that can manage Bazel's output tree (bazel-out/) with a userspace filesystem. We are also defining a gRPC protocol for the communication between Bazel and the service.

Please check the design doc for details. All feedback is welcome!

Related to #12823.

[sluongng]

From reading the design doc, and the comments from reviewers, I wonder if this could be attached with a high-level overview on what are the goals we are trying to solve with this RPC. Then perhaps we could break the goals down into a multi-stage roadmap: What goals will be addressed in the first stage, what will be left for a later stage, and what will NOT be addressed (aka. non-goal)?

If we could break the RPC feature down categorically, then folks could rest assured that some of their needs could be addressed at a later stage. 🤔

[EdSchouten]

I just merged these changes on the Buildbarn side to use the new protocol:

• buildbarn/bb-remote-execution@94b3776
• buildbarn/bb-clientd@ad659bd

I have only been able to test this through unit tests. Not by actually using it. The reason obviously being that I don't have a copy of Bazel that implements the new protocol. I will patiently wait for @coeuvre to post his patch set for Bazel. I also invite others to give the latest version of bb_clientd a try by then.

[coeuvre]

A workable client is available in my branch https://github.com/coeuvre/bazel/tree/remote_output_service_head. I still need time to go through the review process and submit it but you can already try it now. Note that the feature is still pre-alpha so expect bugs and issues.

Thanks @EdSchouten for updating bb_clientd to the new protocol so I can use it as the server during the development.

To try it:

1. Use a bazel binary compiled from https://github.com/coeuvre/bazel/tree/remote_output_service_head
2. Setup bb_clientd
3. Set --remote_cache or --remote_executor as usual.
4. Set --experimental_remote_output_service and --experimental_remote_output_service_output_path_prefix.

[EdSchouten]

Yes, unfortunately it still does. Sonoma 14.4 includes many improvements to the NFS client driver that are of use on the worker side. It makes bb_worker's virtual output directory work (almost) flawlessly. However, it still only implements NFSv4.0, which isn't sufficient.

For Remote Output Service to work well using NFS, we need:

• NFSv4.1/4.2's support for parallel OPEN calls by a single open-owner. With NFSv4.0, all OPEN calls issued against the NFS mount are essentially serialised, which is very bad for performance.
• NFSv4.1/4.2's support for sending callbacks in case directories are modified (CB_NOTIFY). Otherwise macOS won't be notified when StageArtifacts() mutates the file system behind the scenes.

[scosenza]

@EdSchouten For some background (which I'm sure you've heard many times before), our engineers all have Mac laptops, so I was looking into lazy file loading to avoid eagerly downloading files over home internet connections. And while ROS can also help our builds from Linux clients, our Linux machines are already in the datacenter, so network speeds are much less of an issue there. Either way, thanks again for all of your work in this area!

[coeuvre]

• NFSv4.1/4.2's support for sending callbacks in case directories are modified (CB_NOTIFY). Otherwise macOS won't be notified when StageArtifacts() mutates the file system behind the scenes.

I assume this is for StartBuildResponse.initial_output_path_contents? For NFSv4.0, can we simply don't track the modifications and let Bazel figure out which output file has been modified? This might be slower but the overall UX should be better than not using remote output service at all.

[EdSchouten]

@coeuvre It's unrelated to that feature. Think of it like this:

1. Bazel stat()s a file in bazel-out/ and notices it's absent.
2. Bazel kicks off a remote action to get that file built.
3. Bazel will call StageArtifacts() to create that file in bazel-out/.

If Bazel (or a locally running build action) would attempt to stat() that file not long afterwards, they might still get a file-not-found, because the kernel caches the fact that the file does not exist. Or even worse, if the file did exist but it had to be rebuilt, stat() would see the old file. This is because the file system's contents changed behind the kernel's back.

Right now we solve that in bb_clientd by setting mount options acdirmin=0,acdirmax=0 to set caching of directory contents to zero seconds, but this is obviously very bad for performance. And I think that even in that case it doesn't rule out some false caching. For it to work reliably, we need a mechanism where bb_clientd can tell the kernel to purge the name cache entry.

[moroten]

A build of the release-7.2.0 branch, as of 2024-04-16, with cherry-picks from remote_output_service_head is available at https://github.com/meroton/bazel/releases/tag/7.2.0-pre20240416ros.

[coeuvre]

All changes have been cherry-picked to 7.2 branch. You can try it out with the upcoming 7.2 release!

[adincebic]

@buchgr is this on the roadmap for the bazel-remote?

[brentleyjones]

@mostynb is the maintainer of bazel-remote now.

[mostynb]

This is not currently on bazel-remote's roadmap. But it might be possible for other implementations of the fuse service to use bazel-remote as the cache service.

[EdSchouten]

Exactly. It should be possible to use bb_clientd in combination with bazel-remote.