Heuristic:Vespa engine Vespa Maven Parallel Build Optimization
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Build_System |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Build parallelism strategy using 2/3 of available CPU cores for Maven and C++ builds, with heap memory scaled at 1GB per thread and full core utilization for install/test phases.
Description
The Vespa build system uses a tiered parallelism strategy. Compilation phases (Maven and C++) are limited to 2/3 of available CPU cores to prevent resource exhaustion and maintain system responsiveness. Install and test phases use the full CPU count since they are I/O-bound or run independent tests. Maven heap is scaled dynamically: minimum 1GB with maximum set to 1GB per allocated thread. PR builds skip source and javadoc JAR generation for faster feedback.
Usage
Apply this heuristic when configuring Maven builds or C++ compilation for Vespa, or when troubleshooting build performance. The 2/3 rule prevents out-of-memory kills and system unresponsiveness during large parallel builds.
The Insight (Rule of Thumb)
- Action 1: Set Maven thread count to 2/3 of
nproc:NUM_MVN_THREADS = nproc * 2/3. - Action 2: Set C++ build thread count identically:
NUM_CPP_THREADS = nproc * 2/3. - Action 3: Set Maven heap to
-Xms1g -Xmx$(NUM_MVN_THREADS)g(1GB per thread). - Action 4: Use full
nprocfor install and test phases (I/O-bound, independent work). - Action 5: For PR builds, add
-Dmaven.source.skip=true -Dmaven.javadoc.skip=true. - Action 6: Use
--batch-mode --no-snapshot-updatesto avoid interactive prompts and unnecessary remote checks. - Trade-off: 2/3 allocation is slightly slower than full core usage but prevents OOM kills and keeps system responsive.
Reasoning
Large Maven builds with many modules can consume significant memory per thread. Using all cores would require proportionally more heap, risking OOM kills. The 2/3 allocation provides a balance: on a 12-core system, 8 threads with 8GB max heap is aggressive but safe. Install phases (file copying) and test phases (independent test processes) can safely use all cores without the same memory pressure. Skipping source/javadoc JARs in PR builds eliminates unnecessary work when the goal is just validation.
Code Evidence
Thread allocation from .buildkite/Makefile:
export NUM_CPU_LIMIT ?= $(shell nproc)
export NUM_CPP_THREADS := $(shell echo $$(( $(NUM_CPU_LIMIT)*2/3 )))
export NUM_MVN_THREADS := $(shell echo $$(( $(NUM_CPU_LIMIT)*2/3 )))
export MAVEN_OPTS ?= -Xms1g -Xmx$(NUM_MVN_THREADS)g
Maven execution from .buildkite/java.sh:
./mvnw -T "$NUM_MVN_THREADS" "${MVN_EXTRA_OPTS[@]}" "$VESPA_MAVEN_TARGET"
C++ build from .buildkite/cpp.sh:
make -j "$NUM_CPP_THREADS"
Full-core install from .buildkite/install.sh:
make -j "$NUM_CPU_LIMIT" install DESTDIR="$WORKDIR/vespa-install"
Bootstrap single-thread-per-core from bootstrap.sh:128:
mvn_install -am -T1C -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dmaven.source.skip=true