What's better than a genrule?

Alex Eagle's photo
Alex Eagle
·Jun 17, 2022·

4 min read

Subscribe to our newsletter and never miss any upcoming articles

Bazel is pretty confusing. One confusion I've seen a lot is, when do I need to write a custom rule? When can I use a macro? When can I use a genrule?

To make matters worse, Xooglers tend to have a biased answer to this question because of what they saw in google3. Because Google doesn't really use third-party tooling much, there was little usage of genrule to run tools. In a BazelCon talk I once said "we all know genrule is the bestrule" and the Bazel TL gave me a skeptical look.

While the Bazel user guide and user manual preach the benefits of giving Bazel full control over your build process by rewriting all build processes using Bazel-native rulesets (as Google reportedly does internally), this is an immense amount of work. stevenengelhardt.com/2020/10/21/practical-b..

Also the API is kinda lame, and Google hires "only the smartest engineers" so it's not considered a burden to tell someone they must learn Starlark and Bazel's obscure analysis-loading-execution phase semantics.

In reality, genrule semantics are what you want.

  • Point to an existing binary
  • say what command line flags it requires
  • list inputs, outputs, and dependencies

This should be doable entirely in a BUILD file by a regular developer who isn't interested in a half-day meander through build system internals.

Yeah but genrule is not great?

Sadly, the "canonical" genrule bazel.build/reference/be/general#genrule is pretty lacking, and requires that you give it a bash one-liner or script.

run_binary from bazel-skylib is a little better since it supports cmd.exe on Windows and has a smaller sane API. run_binary from Aspect bazel-lib is even better: it can output directories, and supports custom progress messages, mnemonics (bazel's action tagging system) and execution requirements (hints about how to spawn).

The best genrule ought to provide a bunch more features. We have one for rules_js users js_run_binary which can do more things:

  • collect stdout or stderr as output files (great for stubborn tools that insist on putting outputs on stdio when Bazel requires files for everything)
  • intercept the exit code and write to an output file (Bazel will immediately fail if any action exits non-zero, even if the tool is just being cute and returning information via exit code when successful)
  • allow both output files and output directories
  • chdir to a different folder at the start of the action. Some tools expect users to cd into a folder containing some config file and run there.
  • throw away logspam when the action succeeds

Generated Genrules

Okay so if you have a great "genrule" API, the next logical step is for the package manager rules to get involved. Those rules read the metadata for third-party packages, and can see which ones provide developer tool "binaries".

rules_python, rules_nodejs, and rules_js all support auto-generated rules for these. For example if you just wanted to use yamllint in your BUILD file, you can do it like this:

github.com/bazelbuild/rules_python/blob/mai..

load("@pypi//:requirements.bzl", "entry_point")

alias(
    name = "yamllint",
    actual = entry_point("yamllint"),
)

This gives you a genrule-like API for calling the tool. Even better, there's no eager fetch: loading this BUILD file won't make Bazel download the yamllint wheel from pypi. (Same for rules_js generated bins)

@pypi_yamllint//:rules_python_wheel_entry_point_yamllint is what this alias points to, and that's a regular py_binary rule which could be used with the aspect_bazel_lib run_binary above, and you're already done.

So, genrule, macro, or custom rule?

So far everything I showed is just genrule. But it can get unwieldy for developers to directly call the CLI of third-party tooling from the BUILD file. It's a pretty nice pattern to wrap these generated bin entry points with a macro, which is just a preprocessor definition Bazel will resolve early (in the loading phase). When you run bazel query, you're looking at BUILD files after macros expand, so they're really just a bit of syntax sugar.

Be careful not to make a mess with macros - they are leaky abstractions. Read docs.bazel.build/versions/main/skylark/bzl-.. before you write your first one.

Now users BUILD files look just the way you'd wish they did. We still didn't write custom rules. If you've gotten this far, here's my main takeaway:

You should rarely need to write a custom rule

Here are some good reasons to write your own custom Bazel rule:

  • You need to interoperate with other rules using richer information than the built-in providers like DefaultInfo and OutputGroupInfo provide.
  • The underlying tool is too slow on a cold start (ahem JVM and Node applications), and everyone works around this with "watch mode", so you need to write a "Bazel persistent worker" binary and wrap that as a rule.
  • You want to write a toolchain that is platform-aware, to manage Bazel's fetching that tool or to cross-compile for the target platform.
  • The actions to run depend on what outputs a user requests. For example, you can run a quicker transpiler for TypeScript if the user only wants JavaScript outputs and not "declaration" interface files.
 
Share this