Fixing Bazel out-of-memory problems

Photo by Daniela Paola Alchapar on Unsplash

Fixing Bazel out-of-memory problems

Alex Eagle's photo
Alex Eagle
ยทAug 20, 2022ยท

4 min read

Subscribe to our newsletter and never miss any upcoming articles

Memory management is a generally hard topic in computer systems operations. Debugging it inside a cloud-hosted build system is even worse!

There are two potential problems:

  • The Bazel server runs in a JVM, and it internally tries to allocate more objects than the max heap size its allowed.
  • Bazel spawns subprocesses (called "actions", including test actions) and they collectively exhaust the memory in the machine or VM that Bazel runs in.

I'll cover these scenarios separately since they're mostly unrelated.

Of course, the Bazel JVM heap does occupy system memory, so they're related in the sense that a smaller Bazel server footprint would allow for more actions to run, but I've never considered that to be a potential remediation.

Bazel server out-of-memory

How to tell this is happening:

  • Bazel exits with code 33 (see ExitCodes.java)
  • Check the output of bazel info | grep heap if the Bazel server is still running, see if it is near the max.

Some things you can do about it:

System out-of-memory

Bazel schedules actions (build steps and test runners) based on the amount of system resources it thinks are available, and using some heuristic about how much RAM a typical action requires. Two kinds of things can go wrong, either Bazel thinks more RAM is available than the system actually has free, or Bazel underestimates the resources to be reserved for a given action.

By default, Bazel's max concurrency is based on the heuristic that each action needs one CPU core, so the --jobs flag default is the number of (maybe virtual) CPUs on the machine. Note that Bazel reports progress with a "X running" indicator which might lead you to believe that the concurrency is actually higher, but that's a misleading message because it can include actions that are queued waiting for resources

Did you know when @bazelbuild prints progress like
[12 / 100] 32 actions, 30 running
That "running" count includes "remote-cache" spawns! If you have --jobs=16 (the default on 16 core) the other 14 of them aren't actually running, they're queued for "local" spawn. https://t.co/a3w7TWZVTM

โ€” Alex ๐Ÿฆ… Eagle (@Jakeherringbone) August 17, 2022

How to know this is happening:

  • The Bazel server gets killed by the operating system like Bazel server terminated abruptly (error code: 14, error message: 'Socket Closed', log file: ...)

Some things you can do about it:

  • If Bazel is running inside a container, it may calculate available RAM based on the host system rather than what is allocated for the container, due to Default local resources should respect cgroup limits on Linux. The remediation is to explicitly tell Bazel's scheduler what the container limits are by setting the --local_ram_resources flag to match the container runtime.
  • Reduce --jobs so that fewer things run concurrently. This is a blunt approach and makes all builds take longer, but saves you the effort of figuring out which actions didn't get enough resource reservation.
  • Figure out which actions consume a lot of RAM, and tell Bazel's scheduler to reserve more resources for them. For tests, use the test_size attribute - a larger size gets more reserved memory per the table in that documentation. For build actions, the Bazel team recommends using execution properties though to be honest, that looks really complex and I haven't used it myself.
  • Try out the --experimental_local_memory_estimate flag to make Bazel smarter about knowing the available system resources at the time it's scheduling the subprocess to spawn.
  • Investigate using Remote Build Execution so that heavy workloads move off the machine and run on a cloud of executors.
ย 
Share this