Workflow:Vespa engine Vespa CI CD build pipeline
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, Build_Systems, DevOps |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
End-to-end process for building Vespa from source through the Buildkite CI/CD pipeline, producing RPM packages, container images, and signed artifacts.
Description
This workflow describes the complete Vespa build pipeline orchestrated by Buildkite. The process takes the Vespa source repository and produces deployable artifacts through a multi-stage pipeline: version preparation, Java bootstrap and Maven build, CMake configuration and C++ compilation, RPM package creation, Docker container image building, and cryptographically signed artifact publishing. The pipeline supports both x86_64 and ARM architectures, optional sanitizer builds, and integrates with the Vespa Factory release management system.
Usage
Execute this workflow when building Vespa from source for development, testing, or release purposes. This is the primary build path triggered by commits to the master branch (producing releases Monday through Thursday) and by pull requests (producing test builds). Use this to understand the complete dependency chain from source checkout to deployable container images and RPM packages.
Execution Steps
Step 1: Version preparation
Update version strings across all Maven POM files to reflect the target build version. The prepare script replaces SNAPSHOT version strings in all POM files using sed transformations and creates the artifact staging directory structure organized by architecture (x86_64, arm64). This ensures all built artifacts carry consistent version metadata.
Key considerations:
- Version replacement targets all Maven POM files in the repository
- Artifact directories are created at WORKDIR/artifacts/ARCH/rpms and maven-repo
- The VESPA_VERSION environment variable drives all version references
- This step must complete before any compilation begins
Step 2: Java bootstrap and Maven build
Bootstrap the Java build environment by building foundational Maven plugins, parent POMs, and dependency version declarations first. The bootstrap script uses a Maven wrapper (version 3.9.12) to build plugins separately before other modules. Then the full Maven build compiles all Java modules with parallel thread support, producing JARs, test artifacts, and a local Maven repository.
Key considerations:
- Maven plugins must be built before other modules (bootstrap dependency)
- Full bootstrap mode also builds jrt, linguistics, and messagebus for C++ test integration
- Test JARs are collected and staged for C++ integration tests
- The Maven build respects NUM_MVN_THREADS for parallelization
Step 3: CMake configuration
Configure the C++ build environment using CMake with project-specific options. The configuration step sources the GCC toolset, sets the dependency path (/opt/vespa-deps/bin), and generates build files with options for privileged mode, Valgrind unit tests, and optional sanitizer support (ASan, UBSan, TSan). Valgrind is enabled by default but disabled for pull request and sanitizer builds.
Key considerations:
- GCC toolset is sourced from /etc/profile.d/enable-gcc-toolset.sh
- VESPA_USE_SANITIZER enables address, undefined behavior, or thread sanitizers
- ccache is disabled when sanitizers are active
- The build directory is separate from the source directory
Step 4: C++ compilation
Compile all C++ components using Make with parallel thread support. This produces the core Vespa binaries including the proton search engine, storage nodes, config server components, and all supporting libraries. The compilation links against pre-built dependencies at /opt/vespa-deps.
Key considerations:
- Parallelism is controlled by NUM_CPP_THREADS
- System resource limits are reported via show-limits.sh for diagnostics
- The build produces shared libraries, executables, and test binaries
- Dependencies include vespalib, fnet, messagebus C++ implementations
Step 5: RPM package creation
Create installable RPM packages from the compiled artifacts. First, a source RPM is generated using the repository's COPR Makefile. Then rpmbuild rebuilds the source RPM into binary RPMs with zstd compression and custom install paths. Finally, createrepo generates YUM repository metadata over the produced RPMs.
Key considerations:
- Source RPM creation uses make -f .copr/Makefile srpm
- Binary RPMs use zstd compression (w10T4.zstdio payload)
- Debug source packages are suppressed to reduce artifact size
- The YUM repository metadata enables direct installation from the artifact directory
Step 6: Container image building
Build Docker container images for both the Vespa runtime (preview) and system testing. The preview container clones the vespa-engine/docker-image repository and builds using the produced RPMs as the Vespa installation source. The system test container clones vespa-engine/system-test and packages it with the Maven repository for offline test execution. Both containers are tagged with the build version and architecture.
Key considerations:
- Preview container tag: ghcr.io/vespa-engine/vespa-preview-ARCH:VERSION
- System test container tag: docker.io/vespaengine/vespa-systemtest-preview-ARCH:VERSION
- Base images vary by OS version (el8 or el9)
- The build context includes RPMs, Maven repository, and source archives
Step 7: Artifact signing and publishing
Create compressed archives of RPM and Maven repositories, then cryptographically sign all artifacts using Cosign with Buildkite OIDC provider authentication. Each artifact produces a signature (.sig) and certificate (.pem) file. All artifacts are uploaded to cloud storage via the Buildkite agent API and optionally registered with the Vespa Factory release management system.
Key considerations:
- Cosign uses buildkite-agent as the OIDC provider for keyless signing
- Artifacts include RPM repo tarball, Maven repo tarball, and config-model-fat RPM
- Factory API integration uses Athenz mutual TLS authentication
- Build status, release records, and job run metadata are reported to Factory