April 15, 2024
Qing Yang
The Airbnb Tech Blog

How Airbnb achieved a easy and clear migration from Buck to Bazel on iOS, with minimal interference to developer workflows

By: Qing Yang, Andy Bartholomew

At Airbnb, we’re dedicated to offering the perfect expertise for our engineers. To supply a cohesive and environment friendly construct expertise throughout all platforms, we’ve determined to undertake Bazel as our construct system. Bazel is a sturdy construct system extensively utilized within the trade. In alignment with Airbnb’s tech initiatives, each our backend and frontend groups initiated the migration course of to Bazel. Within the first Bazel put up, we begin with our iOS improvement migrating from Buck to Bazel.

We’ll describe the migration strategy which concerned two fundamental items of labor: migrating the construct configuration and migrating the IDE integration. Such a transition can doubtlessly disrupt engineers’ workflows or hinder the event of recent options, however we have been capable of efficiently migrate them with out disrupting the day-to-day developer expertise. Our intention is to assist others who’re presently present process or planning an identical migration.

On the subject of construct configuration, Buck and Bazel exhibit important similarities. They share a comparable listing construction, make use of comparable command line invocation, and, importantly, each make the most of the Starlark language. These similarities current a chance for configuration sharing between the 2 construct methods. This might enable us to reuse our Buck configurations in Bazel, whereas avoiding slowdowns through the “overlap” part once we have been within the technique of migrating and nonetheless actively utilizing each construct methods.

Sadly, there’s a serious drawback: Buck and Bazel make use of distinct guidelines with totally different parameters. As an example, Buck affords guidelines comparable to apple_library and apple_binary, whereas Bazel, relying on the exterior rule units, options guidelines like swift_library and apple_framework. Even in circumstances the place the 2 methods have guidelines with the identical identify, comparable to genrule, the syntax for configuring these guidelines is usually unalike. The totally different design philosophies of those two methods end in varied incompatibilities as properly. As an example, Bazel doesn’t have the read_config operate to learn command line choices in a macro.

Hiding the Variations with rules_shim

After conducting an in-depth evaluation of each Buck and Bazel, we devised a complete structure for the construct configuration to leverage the similarities and tackle the variations between every system.

The construct configuration layers

On the core of this structure lies the rules_shim layer, which introduces two units of guidelines: one for Buck and one other for Bazel. These rule units act as wrappers across the native and exterior guidelines, providing unified interfaces to the layers above.

How does rules_shim work, precisely? By making use of native repositories, we will level the rules_shim repository to totally different implementations relying on the construct system.

That is what the outcome seems to be like in Buck’s .buckconfig:

[repositories]
rules_shim = rules_shim/buck

[buildfile]
identify = BUILD

Observe that we’ve additionally configured Buck to make use of BUILD because the config file, and renamed the present BUCK recordsdata to BUILD, so the identical configuration may be acknowledged by each Buck and Bazel.

In Bazel’s WORKSPACE, we do the next:

local_repository(
identify = "rules_shim",
path = "rules_shim/bazel"
)

In an everyday BUILD file, we use my_library to wrap across the native guidelines and supply the identical interface for every software:

load("@rules_shim//:defs.bzl", "my_library", …)

The app-specific guidelines layer solely must know the interface, not the implementation. In consequence, each time we execute Buck or Bazel instructions, the construct system is ready to retrieve the corresponding implementation from the rules_shim layer. A notable benefit of this design is that we will simply take away the rules_shim/buck after the migration.

Unifying the genrule interface

Inside our iOS codebase, we closely depend on generated code to handle boilerplate and scale back the upkeep burden for engineers. Given the totally different syntax for genrule scripts between the 2 construct methods, we additionally designed a unified interface for genrule. In consequence, the identical genrule script can operate throughout each construct methods. As you might have guessed, the conversion course of is carried out within the rules_shim layer.

We designed the predefined variables within the unified genrule interface.

Changing read_config with choose

Conditional configuration is unavoidable, as a result of there are all the time totally different variants of a constructed product, comparable to debug builds and launch builds. Buck gives a operate known as read_config that reads command line choices in a macro, whereas Bazel doesn’t have this as a result of system’s strict separation of loading phase. It’s value noting that Buck does assist the select operate, though it’s undocumented. We now have migrated all situations of read_config to choose-based situations.

deps = choose(
"//:DebugBuild": non_production_deps,
"//:ReleaseBuild": [],
# SELECT_DEFAULT is outlined in rules_shim to accommodate
# the totally different default strings utilized by Buck and Bazel
SELECT_DEFAULT: non_production_deps,
),

Total, this design achieved the utilization of a single construct configuration for each construct methods, with minimal adjustments to our BUILD recordsdata themselves. In follow, iOS engineers at Airbnb not often have to manually modify BUILD recordsdata, that are mechanically up to date from an evaluation of the underlying supply code. Nonetheless, in circumstances the place it does happen, they will depend on the unified interface while not having to concentrate on the precise underlying construct system.

iOS Engineers at Airbnb primarily work together with the construct system via Xcode. Since first adopting Buck, we now have been using Buck-generated Xcode workspaces for native improvement. Over time, we’ve developed varied productivity-boosting options on prime of this setup, together with the Dev App, a small improvement software centered on a single module; Buck Native, which makes use of Buck as a substitute of Xcode for constructing and leverages distant cache; and Focus Xcode workspace, which considerably improves IDE efficiency by loading solely the modules being labored on.

Within the Bazel ecosystem, a number of options exist for producing Xcode workspaces. Nonetheless, on the time of our analysis, none of them absolutely met our necessities. Moreover, any IDE integration must assist not solely constructing, but additionally enhancing, indexing, testing, and debugging. Given the confirmed observe report and stability of our present workspace setup, we deemed the chance of adopting a totally new one to be exceedingly excessive. Therefore, we determined to develop our personal generator to create a workspace near our current setup. We selected XcodeGen, a well-liked device on this space, as a result of it generates Xcode tasks from a YAML configuration, serving as an abstraction layer to separate the construct system implementation particulars.

The movement of producing the Xcode undertaking

We carried out this migration course of in three phases.

Firstly, we utilized buck question to collect all the mandatory info from the codebase and generate an Xcode workspace, changing the buck project command. This new workspace invoked buck construct through the construct course of. By protecting the construct system unchanged, we have been in a position to make sure compatibility and consider the efficiency of the brand new generator.

Secondly, we carried out a parallel implementation in Bazel utilizing bazel question and bazel construct, incorporating a easy --bazel possibility within the era script that allows switching between the 2 construct methods inside Xcode. Other than the construct system, the consumer interface remained equivalent, making certain that every one IDE operations continued to operate as earlier than.

Lastly, after a adequate variety of customers opted for Bazel and all Bazel-powered options underwent in depth testing, we made the --bazel possibility the default, ending for a easy transition to Bazel. Though we didn’t have to, we might simply roll again if points had occurred. A number of weeks later, we eliminated Buck assist from the generated undertaking.

The tip results of this migration is spectacular. In comparison with the Buck-generated undertaking(buck undertaking), the era time with XcodeGen has been decreased by 60%, and the open time for Xcode has decreased by greater than 70%. In consequence, this new workspace setup acquired prime rankings in an inner developer expertise survey, showcasing the numerous enhancements achieved via this course of.

“All issues in pc science may be solved by one other degree of indirection.” — David Wheeler

Wherever we relied on Buck, we launched a typical interface abstraction and injected separate implementations to deal with the variations between Buck and Bazel. Due to the “indirection” precept, we have been capable of take a look at and replace every implementation with out dramatically rewriting the code, and we efficiently transitioned from Buck to Bazel seamlessly throughout all use circumstances, together with native improvement, CI testing, and releases. The migration course of was executed with out disrupting engineers’ workflows and, in actual fact, allowed us to ship a number of new options, together with SwiftUI Previews assist.

Since Bazel grew to become our iOS construct system, we now have noticed notable enhancements in construct occasions, notably for incremental builds. This shift has enabled us to leverage shared infrastructure, comparable to distant cache, alongside different construct platforms inside Airbnb. Consequently, we now have fostered elevated collaboration throughout platforms.

Migrating our iOS construct system is simply the primary of quite a lot of Bazel migrations underway or accomplished at Airbnb. We now have repos for JVM-based languages (Java/Kotlin/Scala), for JavaScript, and for Go, that are both utilizing Bazel already, or can be sooner or later. We imagine a single construct device throughout our whole codebase will enable us to extra successfully leverage our investments in construct tooling and coaching. Sooner or later, we’ll be sharing classes discovered from these different Bazel migrations.