In https://blog.aspect.dev/releasing-bazel-rulesets I wrote about a pattern we developed for https://github.com/aspect-build/bazel-lib to publish our Go binaries on each release, then use Bazel's toolchains support to fetch those on users machines. This ensures that the Go SDK and libraries are only a development dependency of bazel-lib, not leaked to users.

We recently needed to follow this pattern in https://github.com/aspect-build/rules_py - but it didn't work quite the same way. That's because this time we wrote the tools in Rust, so that we could take advantage of some great OSS work from leaders in the Python ecosystem recently: astral.sh and prefix.dev. In particular, we wanted to use https://docs.rs/rattler_installs_packages/latest/rattler_installs_packages/ to create our virtualenv.

The difference between Go and Rust is in the support for cross-compilation. In Go, it's quite easy for every platform to compile the complete set of release binaries. As a result, we were able to check in the integrity hashes of the binaries at every commit of the repo. If a contributors changes a Go source file, they'll be forced to update the corresponding integrity hashes as part of their PR. As a result, every commit in the git history is "releasable" - the sources include the information needed to safely fetch pre-compiled binaries from the releases page.

Rust is a bit trickier. This is both at the language level, and also because of how rules_rust behaves. We were not able to get all the different cross-compiles working, so the source repo CANNOT always have the integrity hashes. Well... that's okay because we didn't really enjoy having to check those in anyway.

The Fix

After some debate, we formed the requirements:

• The release instructions should still have one step: "push a tag to the repo". This ensures that developers don't push from local machine (non-reproducible, cannot make SLSA attestation that our binaries are built from the tagged sources, etc). It also reduces the maintenance burden which is critical for a small team maintaining lots of rulesets.

• For packaging the ruleset, we're allergic to the complexity of teaching Bazel about a "tree of filegroup targets called release_files as that's too much maintenance work. We prefer just git archive especially because of .gitattributes support for things like "exclude the examples folder from the distribution" and "stamp the release tag into one of the source files".

The solution: the GitHub Actions automation has to build the binaries, then we take those integrity hashes and modify the .tar file produced by git archive to insert them.

Here's a quick code walk-through in case you need to build something similar:

1. https://github.com/aspect-build/rules_py/blob/main/tools/release/BUILD.bazel#L31 is an sh_binary that "delivers" the Rust tools to a $DEST folder

2. https://github.com/aspect-build/rules_py/blob/main/.github/workflows/release.yml#L12-L38 ensures that when a release tag is pushed to the repo, GitHub Actions spins up a machine for each OS to run that "deliver" tool. Then, https://github.com/aspect-build/rules_py/blob/main/.github/workflows/release.yml#L46-L52 we fetch those artifacts to our release job.

3. https://github.com/aspect-build/rules_py/blob/main/.github/workflows/release_prep.sh#L16-L44 The release script runs git archive like usual, but then we modify the tools/integrity.bzl file to contain the new hashes. Note that git archive HAS to be run in a pristine folder so that the release tag ends up stamped as our version in https://github.com/aspect-build/rules_py/blob/main/tools/version.bzl#L3-L5. So we can't do any messing around with files until after we've run it.

4. https://github.com/aspect-build/rules_py/releases/tag/v0.7.1 shows how an automated release looks after I pushed the v0.7.1 tag. Inside the rules_py-v0.7.1.tar.gz artifact, we have all our features:

   1. tools/version.bzl contains _VERSION_PRIVATE = "v0.7.1" thanks to https://github.com/aspect-build/rules_py/blob/78832372c4a8c17259882f9c27715f8ef6cf4451/.gitattributes#L8-L9

   2. the examples folder is absent, to keep the release artifact from getting huge as we pile on more example usage, thanks to https://github.com/aspect-build/rules_py/blob/78832372c4a8c17259882f9c27715f8ef6cf4451/.gitattributes#L5-L6

   3. tools/integrity.bzl contains the result of that release script, i.e.RELEASED_BINARY_INTEGRITY = { "unpack-aarch64-apple-darwin": "12502bad22e0725baeb37b531322fe1d12dd053ae6716bcead88cad5e26f0dab",
...

5. Cool, now we just need to ensure that rules_py users are setup to fetch those binaries rather than need rules_rust. Thanks to the git-stamping in version.bzl, the macro users call can tell whether you're using a prerelease (developing on rules_py or fetching a SHA of the repo: https://github.com/aspect-build/rules_py/blob/78832372c4a8c17259882f9c27715f8ef6cf4451/py/repositories.bzl#L61
   This way users of a release get a different set of toolchains registered.

6. That pre-built binary toolchain is just the usual incantation: one repo that contains toolchain calls for every platform: https://github.com/aspect-build/rules_py/blob/78832372c4a8c17259882f9c27715f8ef6cf4451/py/private/toolchain/repo.bzl and then one repo for each platform that actually does the tool fetching: https://github.com/aspect-build/rules_py/blob/78832372c4a8c17259882f9c27715f8ef6cf4451/py/private/toolchain/tools.bzl. That ensures you only download tools for the platforms Bazel selected in toolchain resolution. Note that one tool is for the exec platform (we unpack wheels in an action) and the other is for the target platform (we create the virtualenv inside the py_binary runtime). Bazel handles this nicely, e.g. if you build a py_image on a Mac, you'll fetch the unpack binary only for darwin_arm64 and the venv binary only for linux_x86.

7. Finally, we've got a test in https://github.com/aspect-build/rules_py/tree/main/e2e/use_release that asserts that the release works as we expected: the pre-built binary toolchains were selected, and we only downloaded tools for the toolchain-resolved platform.

That's a pretty deep-dive into what we do in Bazel rules to provide an awesome end-user experience. More to come on rules_py as we ship our 1.0!