Dealing with limited number of tool licenses

[frefra]

Hello,

We are using Bazel extensively for FPGA builds. These differ from software builds in that there are no rules (so we built everything ourselves) and one specific issue we run into is tool licenses.

Some tools in the build flow have a limited number of licenses (four or less) which are shared among a team of some 50 engineers. These licenses are required in test targets but also in build targets.

This poses two issues:

1. Build and test steps are sometimes queued for license checkout which can lead to timeouts.
2. Parallelism is preferred but not for steps for which we have limited licenses. This causes excessive queuing, not just for automated builds but also for development.

The issues can be mostly solved for test targets by running tests with --jobs 1. For build targets this is more problematic. Ideally there should be a way to limit parallelism for specific target types (rules).

Does anyone know whether this is solvable within Bazel, without invoking builds for different target types with different settings?

Thanks a lot!

[sluongng]

This class of issue is more straightforward to solve with Remote Build Execution(RBE) than using local execution.

With an RBE (such as BuildBuddy, where I work), you could configure a dedicated pool of machines to execute a specific set of actions.

For example, we often have customers with a limited number of dedicated bare-metal hardware shared among X engineers.
These bare-metal hardware are configured as a BuildBuddy's Executor Pool.

Then using Bazel configuration, you could modify a build target to use this pool

cc_binary(
    ...
    exec_properties = {
        "Pool": "special-hardware-pool",
    },
)

If you need to configure the pool on the rule's action level (more fine grain than Bazel's target level), then you could set up each action in a separate execution group that is mapped to exec_properties.Pool.

[tjgq]

Another option is to use --local_extra_resources (I believe you need Bazel 7, though - it hasn't been backported to 6.x).

[sluongng]

local_extra_resources is what I thought about initially when I read this problem as well 🤔.

But I thought about this a bit more: if an org of 50 people have to share 4 licenses, you would not be able to distribute the resource locally to all engineers' laptop. The task scheduling, therefore, would need to be aware of multiple machines instead of a single machine.

The initial solution going into my head was to implement a persistent worker with concurrency == 1 on each developer machine. The persistent worker could talk to a centralized service to queue and lease the license before each execution. However, implementing such a license leasing service and a local persistent worker comes with many challenges and multiple points of failure. It's easier to leverage the existing RBE capability of Bazel to delegate the scheduling to a remote scheduler and remote execution engine. That way, instead of worrying about 50 laptops, you would only need to take care of 4 action executors (workers).

[frefra]

Thank you for your answers, both are interesting ideas to look into.

The BuildBuddy pool suggestion would not fix the whole issue (as we want to run simulations locally while developing the test bench code), but at least it will solve the "daily build rush hour". Currently we ensure these test runs are done with --jobs 1, but that does not prevent a lot of separate builds to be started on different machines.

I also tried the --local_extra_resources option, which works well but unfortunately only for tests, not for regular build actions.

Although sharing 4 licenses with 50 engineers (and a large number of build servers) sounds cumbersome, in reality the problem is not that big. Queuing in itself is not an issue since each run takes a limited amount of time. Wait times are rarely more than a few tens of seconds. Most issues occur during nightly builds when many build servers claim licenses at once. But it seems this can be completely solved with a sophisticated RBE solution.

Thanks again for your insights!