Moving TypeScript code into a Bazel monorepo

note, this article is still a work-in-progress as of September 2022

Aspect's rules_js and rules_ts finally make Bazel work well with TypeScript projects. If you're a Dev Infra team who's ready to do a migration, here's a practical step-by-step guide.

1. Planning

Sorry, I know you want to get right into the code. However, in a larger organization there are some things you may trip on later.

  1. Research your options. If you're a Bazel enthusiast, chances are that some of your co-workers are not, so you'll want to be prepared to justify your choices. https://monorepo.tools is one nice resource for comparing across the frontend ecosystem.
  2. Bazel introduces risk. It's not the most popular build tool for TypeScript. You might want to budget for OSS support in case you need help, which we offer: https://www.aspect.dev/services#support
  3. Like https://rushjs.io, rules_js uses pnpm under the hood. rules_js does support dynamically importing npm or yarn lockfiles, however it adds complexity, a performance hit, and some long-tail bugs both in how it's implemented and how developers might get different results outside of Bazel. Will you change to pnpm as a "pre-factoring" before Bazel? Will you re-train your developers to run pnpm rather than npm or yarn?
  4. Skim all the documentation, so that you at least have a sense of where to search when you have questions. I won't try to repeat it here.
  5. Is it time yet? You might want to do a small-scale Proof-of-Concept and dry-run the migration with a friendly team first, to see what's missing. For example, Aspect is working on auto-configuration of BUILD files (as a JS extension for Gazelle) so if your team strongly dislikes editing these by hand, you might need to wait.
  6. Who will own the monorepo? If you never thought about the term "governance" as applied to source code, you really need to do so now. There will be decisions about code consistency, policies like rollbacks, how to keep it green, etc. If you have no Dev Infra team, you may need to do some political work first to staff a couple of positions so that this is actually someone's job. Once your monorepo becomes a "tragedy of the commons", it's very hard to repair.

2. Setup TS monorepo

Your DevInfra team or "governance group" should be closely involved in decision-making.

Follow the READMEs from rules_ts, or look in Bazel examples. Here are some things to look out for:

  • Probably use workspaces to make a dependency graph among the first-party npm packages that will make up your monorepo. You'll want developers to stay comfortable with their existing configurations, which are likely in package.json files.
  • Dependencies should always stay local. If some TS code in package foo depends on pkg-a, then that dependency should stay in foo/package.json. (Note that Google has a "single version policy" and you can too - rules_js removes the performance penalty that rules_nodejs had when all dependencies appear in a root /package.json. However it shifts the burden for upgrades to one engineer applying to the whole monorepo. That's a benefit to the organization but a cost to that developer and her project manager who just wants to ship features, so it's a political "hot potato".)
  • The root BUILD file will have the npm_link_all_packages() call to create the node_modules tree under bazel-out. Nested npm packages should make exactly the same call to create their nested node_modules trees.
  • Setup code formatting with prettier. To make it consistent for everyone and to work across languages, we suggest using https://github.com/aspect-build/bazel-super-formatter

3. "Slurp" commits to the monorepo

You want to minimize disruption as you move code from many repositories into the monorepo.

First, make sure the monorepo is a good developer experience. You don't want to create detractors who decide they hate Bazel and will never support your migration.

To actually move the code, you should be careful to preserve git history. Let's say you have a repo named other-repo, and we want to bring all of its commits into the monorepo under a folder called other. First install git-filter-repo, then run it like so:

$ cd other-repo

# Make sure you are carrying the current history of the default branch, don't lose anything!
other-repo$ git fetch; git reset --hard origin/main

# Rewrite the history as if it always had been authored under the other/ folder
other-repo$ git filter-repo --to-subdirectory=other

# make sure you're at HEAD of monorepo as well
$ cd ../mono-repo
$ git fetch; git reset --hard origin/main

# Make the other-repo history visible to the monorepo, 
mono-repo$ git remote add other ../other-repo
mono-repo$ git fetch other

# Make a "merge" commit that has one parent at monorepo HEAD and another at other-repo rewritten HEAD
mono-repo$ git merge other/main --allow-unrelated-histories

At this point you can test as usual and send the merge commit as a PR. Note that you might have to change the monorepo branch protection to allow merge commits, if you normally permit only linear history. At this point, you CANNOT REBASE. If you need to make changes, just do these steps again.

As soon as you land this merge commit to the monorepo, you're in danger of the code diverging. You should use whatever means you have to avoid new commits landing in other-repo, as you'll need another recipe to rewrite-and-sync those commits and they might not merge cleanly. As quickly as possible, get the owning team to verify their CI and release workflows, and archive the other-repo. It's easier to un-archive it again later if needed than to deal with consequences from it being a "fork" of the code.

4. Train your developers that code sharing is now a thing

In many-repo workflows, it's a pain to share code. As an application developer, you have to bump the version of your dependencies, deal with conflicts between libraries, and many other things. As a library owner, you have users of old versions who post bugs, but don't want to move to latest.

Monorepo isn't just putting the code together, it should also mean you move to trunk-based development. Dependencies are "at HEAD" and library changes take effect in applications immediately. This sounds scary, but that's what "Continuous Integration" means. You may need to update projects as you bring them into the repo so that they stop fetching your first-party library code from Artifactory or npm, and instead reference the local copy in the monorepo.

In particular, make sure that the Bazel dependency graph makes sense, and that application developers can easily re-use packages across the repository. Be prepared to govern! For example, use Bazel's package visibility to enforce SLAs - a casually-developed (or abandonware) library shouldn't permit dependencies from a business-critical applications.

5. Setup CI/CD workflows

Bazel is supposed to make your Build and Test fast. However it has setup steps (Repository rule execution and Analysis phase over the target graph) and these always run before any cache hits can be looked up. Therefore a cold Bazel worker may be slower than your legacy build.

We suggest looking at our new Aspect Workflows product and read our other blog posts about making Bazel fast on CI.