Principle:Vespa engine Vespa CPP Compilation

Overview

C++ compilation transforms source code into machine code through preprocessing, compilation, and linking. Parallel compilation with make -j distributes work across CPU cores. Vespa requires C++20 with GCC 12+ or Clang 16+. In the CI/CD pipeline, this stage takes the Makefiles generated by CMake configuration and produces the native binaries that form the core of the Vespa search and storage engine.

Motivation

Vespa's C++ codebase implements the performance-critical components of the system: the search core, document storage, tensor evaluation, and network transport layers. These components are written in C++ for maximum runtime performance and low-level system control. The compilation stage is one of the most resource-intensive parts of the build pipeline, consuming significant CPU time and memory.

Key challenges addressed by the compilation stage:

Build parallelism: The Vespa C++ codebase contains thousands of source files. Parallel compilation is essential to keep build times within acceptable bounds.
Compiler requirements: C++20 features such as concepts, coroutines, and ranges are used throughout the codebase, requiring a modern compiler.
Dependency correctness: The build system must correctly track header dependencies to ensure that changes to a header trigger recompilation of all affected translation units.

How It Works

The Compilation Pipeline

Each C++ source file goes through a multi-stage compilation process:

Preprocessing: The C preprocessor expands #include directives, macros, and conditional compilation blocks. For Vespa, this can expand a single source file from hundreds of lines to tens of thousands of lines due to deep include hierarchies.
Compilation: The compiler translates preprocessed C++ source into assembly code, performing syntax checking, type checking, template instantiation, and optimization.
Assembly: The assembler converts assembly code into machine code, producing object files (.o).
Linking: The linker combines object files and libraries into shared libraries (.so) and executables.

Parallel Compilation

The build uses GNU Make's parallel execution feature:

make -j "$NUM_CPP_THREADS"

The -j flag specifies the maximum number of concurrent compilation jobs. The value of NUM_CPP_THREADS is typically set to the number of available CPU cores or slightly higher, as some compilation jobs are I/O-bound and others are CPU-bound.

Make uses the dependency graph generated by CMake to determine which targets can be built in parallel. Two targets can be built concurrently if neither depends on the other. This is the primary mechanism for reducing total build time.

Compiler Requirements

Vespa's C++ code requires:

Requirement	Minimum Version	Reason
C++ Standard	C++20	Concepts, coroutines, ranges, `std::format`
GCC	12+	Full C++20 support
Clang	16+	Full C++20 support (alternative compiler)
Linker	System default	Linking shared libraries and executables

The GCC toolset is activated by sourcing /etc/profile.d/enable-gcc-toolset.sh at the start of the compilation script.

Environment Variables

Variable	Description	Example Value
`NUM_CPP_THREADS`	Number of parallel compilation threads	`16`
`SOURCE_DIR`	Root directory of the Vespa source checkout	`/vespa`

Design Considerations

Thread count tuning: The optimal value for NUM_CPP_THREADS depends on the build machine's CPU count and available memory. Each compiler process can consume 1-2 GB of RAM during template-heavy translation units. Setting the thread count too high can cause out-of-memory failures, while setting it too low wastes CPU resources. The Buildkite pipeline typically uses a value matched to the machine's core count.

ccache integration: When enabled through the CMake configuration stage, ccache intercepts compiler invocations and caches the results. On subsequent builds where source files have not changed, ccache serves the cached object file directly, reducing compilation time from minutes to milliseconds for unchanged files. Ccache is disabled for sanitizer builds to prevent cache pollution.

Incremental builds: Although the CI/CD pipeline typically performs clean builds, the Make-based build system supports incremental builds by tracking file modification times. This is useful during development, where only changed files and their dependents need recompilation.

Resource limit reporting: The compilation script optionally invokes show-limits.sh to log system resource limits before starting the build. This diagnostic output aids in troubleshooting build failures caused by exceeding ulimit values (e.g., too many open files).

Build Outputs

The compilation stage produces:

Shared libraries (.so files): Vespa's native libraries, including the search core, document processing, and network transport.
Executables: Native server processes and command-line utilities.
Static libraries (.a files): Libraries that are statically linked into other targets.
Test binaries: Unit test executables that are run in subsequent test stages.

These outputs are used by the RPM package creation stage to bundle the binaries into installable packages.

Relationship to Other Build Stages

C++ compilation depends on both the Java bootstrap (for JNI headers and test JARs) and CMake configuration (for generated Makefiles):

Java Bootstrap ----+
                   +--> CMake Configuration --> C++ Compilation --> RPM Package Creation
Version Preparation --+

Related Pages

Implemented By

Implementation:Vespa_engine_Vespa_Cpp_Sh

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment