Adopting Bazel's new package manager

Photo by Kimberly Farmer on Unsplash

Adopting Bazel's new package manager

Alex Eagle's photo
Alex Eagle
·Dec 19, 2021·

6 min read

Subscribe to our newsletter and never miss any upcoming articles

Bazel packages (called "modules") have historically been distributed with a long "WORKSPACE snippet", which required users to install and configure the module and also its dependencies. This caused a lot of headache for users, since the first declaration of some dependency wins, and so a wrong order of some transitive initialization code results in baffling errors which can be traced back (often with significant effort) to a root cause of version skew.

As a result, rules authors had to be very careful about taking dependencies. To ensure an easier user experience, rules_nodejs has always refused to add any dependencies at all, even on common modules like bazel-skylib.

Yun and Xùdōng from the Bazel team at Google have been hard at work on fixing this, in a new feature called "bzlmod". The problem has actually had a few false starts in the past (the Bazel "Federation" was one) so it's awesome that this is nearly ready for adoption.

You can read about the design of the feature in the Bzlmod User Guide so I won't repeat that here. Instead we'll just dive into how you can use it!

What it looks like

As an example, I'll use our SWC rule from rules_swc which runs a super-fast JavaScript/TypeScript transpiler written in Rust.

If you use this rule today, there's a complex bunch of code to copy-paste into your project's WORKSPACE file, as illustrated by the install documentation on a release: https://github.com/aspect-build/rules_swc/releases/tag/v0.2.1

Not only is that code that every user of the rule has to maintain, but all there's that headache mentioned before: this code interacts with other install code the user copied into that file. rules_swc depends on a nodejs toolchain, a rust binding, a bunch of npm packages, bazel-skylib, and Aspect's bazel-lib of helper functions. That graph is pretty complex to leak through to the end-user!

Under bzlmod, you can depend on rules_swc with just one line in your MODULE.bazel file:

bazel_dep(name = "aspect_rules_swc", version = "0.2.0")

The example repo shows that an swc target works correctly, using MODULE.bazel to declare the dependencies.

Kicking the tires yourself

First, you'll need to use Bazel 5.0 or greater. As of this writing, you write this in your .bazelversion file:

5.0.0rc3

Follow github.com/bazelbuild/bazel/issues/14013 to find out when 5.0.0 final has shipped.

This should update your local Bazel version. If it doesn't, your install is tied to one version, which you should fix following docs.bazel.build/versions/main/install-baze...

Next you need to opt-in to the bzlmod feature. Add this to your .bazelrc file:

common --experimental_enable_bzlmod

If you run bazel info now, you'll immediately get an error that you must create a MODULE.bazel file in the repository root, next to WORKSPACE. This file contains a syntactic subset Starlark, similar to BUILD.bazel files.

If you want to create your own registry, you'll probably start by forking bazelbuild/bazel-central-registry. You'll then want to use the --registry flag, note that the value has to be a raw github server.

For this example, I'll add Aspect's registry to .bazelrc with the line

common --registry=https://raw.githubusercontent.com/aspect-build/bazel-central-registry/main/

Note, main is a floating reference there. If you're pushing commits to your registry, you'll find that GitHub's CDN has too long of a Time-to-live, and your edits won't show up. Replace main with a commit SHA to fix this.

If you add a dependency to your MODULE.bazel, for example

bazel_dep(name = "aspect_bazel_lib", version = "0.3.0")

and do a build, you'll find there's now an external repository with that name and version in the external folder in your output_base. (Look in $(bazel info output_base)/external)

You can have a mix of dependencies declared in WORKSPACE and in MODULE.bazel. That's nice so you can do the migration one package at a time.

However in practice you'll probably find many modules you depend on which aren't present on the registry at all. You'll have to bug their authors to add a MODULE.bazel file in their repo and publish to the registry. You could also contribute this yourself; see the next section.

As a Rule Author

If you write your own rules, you'll have to interact with bzlmod as a publisher too.

You'll probably start with a local clone of the registry. The bazel server process caches the fetches from the registry (even if you use a file:/// uri, sadly). So while testing your registration, you'll have to bazel shutdown before every bazel command.

In your clone of bazel-central-registry, you can run the wizard ./tools/add_module.py. This will prompt you for the info it needs. Delete any cruft (as of writing, it creates temp .json files in the working directory) and commit and push changes. You can push them to your own fork of the registry before making a PR to the upstream. Note that the BCR presubmit tests will only work when you open that PR.

It's common to need patches on the upstream rule, at least to start with. Create it with a command like git diff > ../bcr/modules/my_module/0.1.0/patches/bzlmod.patch (or git show if the changes are already committed). You'll have to create a subresource integrity hash of the content of that patch, with a command like

shasum -a 256 modules/aspect_rules_js/0.3.0/patches/bzlmod.patch | awk '{print $1}' | xxd -r -p | base64`

Be careful, if the patch file contains edits to the project's WORKSPACE it will fail to apply, because at the time bzlmod applies it, the WORKSPACE file is auto-generated 2 lines of code instead of what came from your rules repo. Then register the patch in your source.json like other examples in BCR.

Things that will need to be patched or changed include:

  • toolchain registration: Extensions can't call native modules (they are evaluated on a different thread for performance). bzlmod has a different syntax for this anyhow, so repository rules shouldn't need to register toolchains. See this patch on rules_sh as the example I followed.
  • fixed repository names: bzlmod maps the repositories in order to namespace them for the strict visibility feature. If your rules expect to create an external repository @foo and then use @foo//some:target in a BUILD file or as a default for an attribute, you'll find that repo doesn't exist. Or in my case I had BUILD files assuming @foo//:foo target would exist, and needed to make it @foo//:pkg so the target is predictable without knowing the repository name.
  • result of one repository rule used by another: github.com/bazelbuild/bazel/issues/14445 - I had to refactor rules not to require this feature

More resources

By default, the registry at https://github.com/bazelbuild/bazel-central-registry is used to discover Bazel modules. Other registries may be created, in particular large companies will likely run a private registry for their own modules.

Yun created a bunch of examples you can reference: https://github.com/meteorcloudy/bzlmod-examples.

Open issues with bzlmod: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+project%3Abazelbuild%2Fbazel%2F9

There's a listing of resources, including the BazelCon talk: https://docs.bazel.build/versions/main/bzlmod.html#external-links

 
Share this