rules_ts benchmarks

Aspect's rules_ts is a port of rules_nodejs's @bazel/typescript package that provides a ts_project rule layered on top of rules_js, Aspect's new high-performance Bazel rule set for JavaScript, designed from the ground up with performance in mind.

The ts_project rule from rules_ts has the same API as its predecessor from @bazel/typescript, but it leaves the gate with performance improvements that were not possible under rules_nodejs.

In this post, we'll compare build times for those two ts_project implementations, as well as against ts_library from @bazel/concatjs (the original TypeScript rule from Google), and against the vanilla TypeScript compiler, tsc.

To learn about how rules_js also makes npm dependencies fast with Bazel check out our rules_js npm benchmarks

Methodology

These benchmarks were run against a generated TypeScript code base of 5 features, 10 modules per feature, 10 components per module and 1001 lines of code per component. This makes for a total of 555 TypeScript files containing 500995 lines of TypeScript code in aggregate.

For the Bazel build, each module maps to one Bazel target for a total of 55 ts_project/ts_library targets.

Configuration

These benchmarks were run on a MacBook Pro (16-inch 2019), 2.4 GHz 8-Core Intel Core i9, 64 GB 2667 MHz DDR4 running macOS Monterey 12.3.1

Versions of typescript and rule sets used were,

Full builds vs. "devserver" builds

In these benchmarks we measure two different scenarios:

  1. A full clean build (bazel build ...) followed by an incremental bazel build ... after making a change to a leaf TypeScript file. This scenario includes type-checking.

  2. A clean "devserver" build (bazel build :devserver), which emulates a typical developer workflow of building while running a tool such as a devserver, followed by an incremental bazel build :devserver after making a change to a leaf TypeScript file.

The "devserver" scenario is an important measure that emulates the typical local development workflow of coding while running tools such as a devserver or a test runner such as jest. These tools are often run in watch mode while making changes to source code. Faster build times on such changes are critical to reduce the round-trip-time to get feedback on those changes. Type-checking is not included, because it's assumed the developer already got such feedback in their editor, and type-checks will be run along with tests in CI.

Ideal build times to maximize developer productivity are less than 1 second on changes to leaf nodes and less than 10 seconds on changes that affect large parts of the graph. Ideally these ideal times are sustained even on large projects.

ts_project vs. ts_library

ts_project was originally developed in rules_nodejs as an alternative to ts_library to provide a cleaner API better suited for the many ways TypeScript is used outside of Google. While the API was better suited for the wild, it could not compete with ts_library on performance, because the latter uses a heavily optimized and deeply integrated wrapper around the TypeScript compiler.

The new ts_project from rules_ts has significantly reduced the performance gap with ts_library by adding first-class support for Bazel workers.

rules_js, which rules_ts is layered on, has made first-class worker support in rules_ts possible by doing away with the dynamic runtime node_modules linking that rules_nodejs uses.

While ts_library can still slightly outpace ts_project with worker mode in full clean build times, in the "devserver" scenario ts_project has a significant advantage over ts_library by allowing you to configure a separate tool for transpiling TS -> JS.

In these benchmarks, we'll measure ts_project configured with swc as the transpiler. swc is an order of magnitude faster than TypeScript for pure transpilation but it does not type check, so TypeScript is still used for type checking in this split configuration.

The split configuration also removes type checking from the build graph for devserver and test targets so only transpilation is needed to build them, reducing the round-trip-time on changes when running such targets by an order of magnitude. Type-checking is handled in separate targets that can be run explicitly or with the catch-all bazel build ....

Results

Without further ado, here are the results of the benchmarks.

Full clean builds

Screen Shot 2022-06-21 at 2.34.15 AM.png

ts_library leads the pack for full (transpilation & type checking) clean build times. It has been heavily optimized inside Google and is integrated deeply with TypeScript compiler internals. The ts_library API, however, is not well suited for the many ways that TypeScript projects are configured outside of Google and it does not integrate well with many other rules and tools in the frontend ecosystem.

rules_ts's ts_project is a competitive runner up. It makes significant performance gains over its predecessor from @bazel/typescript by adding first-class support for worker mode.

This is only our initial pass at worker mode for ts_project and we believe we can optimize it further in the future by taking advantage of Bazel features such as multiplexed workers. Stay tuned for future performance improvements in this rule.

Incremental full builds

Screen Shot 2022-06-21 at 2.34.21 AM.png

All the Bazel rules measured are relatively close in incremental full build times with ts_library taking the lead and ts_project from rules_ts with worker-mode and swc for transpilation runner up. In this benchmark, vanilla tsc is slowest but in smaller projects it can be quite fast.

Clean "devserver" builds

Screen Shot 2022-06-21 at 2.34.26 AM.png

Clean "devserver" builds are where ts_project, configured with swc as the transpiler, really stands out and is an order of magnitude faster than the rest. The 500,000+ lines TypeScript code in this benchmark take even the heavily optimized ts_library more than 40 seconds to build while swc can transpile the same in 3 seconds flat.

swc is fast enough to spawn to be configured as one action per TypeScript file, which means that with remote execution the 555 .ts file targets in this benchmark could be distributed across 555 remote executors and transpile nearly instantly. ts_library, on the other hand, does not split into one target per file so it could parallelize into only 55 actions with remote execution in this benchmark.

Incremental "devserver" builds

Screen Shot 2022-06-21 at 2.34.34 AM.png

On incremental "devserver" builds, where ts_library and even ts_project from @bazel/typescript are fairly fast, ts_project configured with swc for transpilation is still an order of magnitude faster.

The bottom line

With rules_ts, front-end developers can finally get the near instant round-trip-times they are used to with optimized front-end build systems such as Vite with Bazel.

We feel this improvement, along with fast npm dependency management will get more web developers on board with building with Bazel.

Bazel & JavaScript is about to get a whole lot better!