After theWhistler All-Handsthis past summer, I started seriously looking at whether Firefox should switch to usingBazelfor its build system.
The motivation behind switching build systems was twofold. The first motivation was that build times are one of the most visible developer-facing aspects of the build system and everybody appreciates faster builds. What’s less obvious, but equally important, is that making builds faster improves automation: less time waiting for try builds, more flexibility to adjust infrastructure spending, and less turnaround time with automated reviews on patches submitted for review. The second motivation was that our build system is used by exactly one project (ok,two projects), so there’s a lot of onboarding cost both in terms of developers who use the build system and in terms of developers who need to develop the build system. If we could switch to something more off-the-shelf, we could improve the onboarding experience and benefit from work that other parties do with our chosen build system.
You may have several candidates that we should have evaluated instead. We did look at other candidates (although perhaps none so deeply as Bazel), and all of them have various issues that make them unsuitable for a switch. The reasons for rejecting other possibilities fall into two broad categories: not enough platform support (read: Windows support) and unlikely to deliver on making builds faster and / or improving the onboarding / development experience. I’ll cover the projects we looked at in a separate post.
With that in mind, why Bazel?
Bazel advertises itself with the tagline “{Fast, Correct} – Choose two”. What’s sitting behind that tagline is that when building software via, say,Make, it’s very easy to writeMakefiles in such a way that builds are fast, but occasionally (or not-so-occasionally) fail because somebody forgot to specify “to build thing X, you need to have built thing Y ”. The build doesn’t usually fail because thing Y is built before thing X: maybe the scheduling algorithm for parallel execution inmakechooses to build Y first 99. 9% of the time, and 99% of those times, building Y finishes prior to even starting to build X.
The typical solution is to become more conservative in how you build things such that you can be sure that Y is always built before X… but typically by making the dependency implicit by, say, ordering the build commands Just So, and not by actually making the dependency explicit tomakeitself. Maybe specifying the explicit dependency is rather difficult, or maybe somebody just wants to make things work. After several rounds of these kind of fixes, you wind up withMakefiles that are (probably) correct, but probably not as fast as it could be, because you’ve likely serialized build steps that could have been executed in parallel. And untangling such systems to the point that you can properly parallelize things and that you don’t regress correctness can be… challenging.
(I’ve usedmakein the above example because it’s a lowest-common denominator piece of software and because having a concrete example makes differentiating between “the software that runs the build” and “the specification of the build” easier. Saying “the build system” can refer to either one and sometimes it’s not clear from context which is in view. not assume that the problems described above are necessarily specific tomake; the problems can happen no matter what software you rely on.)
Bazel advertises a way out of the quagmire of probably correct specifications for building your software. It does this — at least so far as I understand things, and I’m surethe Internet will come to correct meif I’m wrong— by asking you to explicitly specify dependencies up front. Build commands can then be checked for correctness by executing the commands in a “sandbox” containing only those files specified as dependencies: if you forgot to specify something that was actually needed, the build will fail because the file (s) in question aren ‘ t present.
Having a complete picture of the dependency graph enables faster builds in three different ways. The first is that you can maximally parallelize work across the build. The second is that Bazel comes with built-in facilities forfarming out build tasks to remote machines. Note that all build tasks can be distributed, not just C / C / Rust compilation as viasccache. So even if you don’t have a particularly powerful development machine, you can still pretend that you have a large multi-core system at your disposal. The third is that Bazel also comes with built-in facilities foraggressive caching of build artifacts. Again, like remote execution, this caching applies across all build tasks, not just C / C / Rust compilation. In Firefox development terms, this is Firefox artifact builds done “correctly”: given appropriate setup, your local build would simply download whatever was appropriate for the changes in your current local tree and rebuild the rest.
Having a complete picture of the dependency graph enables a number of other nifty features. Bazel comes witha query language for the dependency graph, enabling you to ask questions like “what jobs need to run given that these files changed? ”This sort of query would be valuable for determining what jobs to run in automation; we have a half-hearted (and hand-updated) version of this in things likefiles-changedin Taskcluster job specifications. But things like “run$ OStests for$ OS– only changes ”or“ run just the mochitest chunk that contains the changed mochitest ”become easy.
It’s worth noting here that we could indeed work towards having the entire build graph available all at once in the current Firefox build system. And we have remote execution and caching abilities via sccache, even moreso now that sccache-dist is being deployed in Mozilla offices. We think we have a reasonable idea of what it would take to work towards Bazel-esque capabilities with our current system; the question at hand is how a switch to Bazel compares to that and whether a switch would be more worthwhile for the health of the Firefox build system over the long term. Future posts are going to explore that question in more detail.
This entry was posted on Monday, October 28 th , 2019 at 1: 11 pm and is filed underUncategorized. You can follow any comments to this entry through theRSS 2.0feed. You canleave a comment, ortrackbackfrom your own site.
GIPHY App Key not set. Please check settings