Implementation:Vespa engine Vespa Bootstrap And Java Sh

CI_CD Build_Systems

Overview

This page documents the implementation of the Vespa Java bootstrap and Maven build scripts: .buildkite/bootstrap.sh, .buildkite/java.sh, and the root-level bootstrap.sh. Together these scripts set up the Maven wrapper, build parent POMs and plugins, compile all Java modules, and collect JAR files needed by C++ tests.

Type: External Tool Doc

Code Reference

.buildkite/bootstrap.sh

#!/usr/bin/env bash
# .buildkite/bootstrap.sh (L1-24)

set -o errexit
set -o nounset
set -o pipefail

if [[ -n "${DEBUG:-}" ]]; then
    set -o xtrace
fi

echo "--- Running Vespa bootstrap"
cd "$SOURCE_DIR"
./bootstrap.sh full

echo "Setting up test JAR directory..."
mkdir -p "$VESPA_CPP_TEST_JARS"

echo "Collecting JAR files for C++ tests..."
# shellcheck disable=2038
find . -type d -name target -exec find {} -mindepth 1 -maxdepth 1 -name "*.jar" \; \
    | xargs -I '{}' cp '{}' "$VESPA_CPP_TEST_JARS"

.buildkite/java.sh

#!/usr/bin/env bash
# .buildkite/java.sh (L1-31)

set -o errexit
set -o nounset
set -o pipefail

if [[ -n "${DEBUG:-}" ]]; then
    set -o xtrace
fi

mydir=${0%/*}
shlim=${mydir}/show-limits.sh
if [ -x "${shlim}" ]; then
    "${shlim}" || echo "failed: ${shlim}"
fi

echo "--- Building Java components"
# shellcheck disable=1091
source /etc/profile.d/enable-gcc-toolset.sh

PATH=/opt/vespa-deps/bin:$PATH

cd "$SOURCE_DIR"

echo "Running Maven build with target: ${VESPA_MAVEN_TARGET} [threads: ${NUM_MVN_THREADS}]"
read -ra MVN_EXTRA_OPTS <<< "$VESPA_MAVEN_EXTRA_OPTS"
./mvnw -T "$NUM_MVN_THREADS" "${MVN_EXTRA_OPTS[@]}" "$VESPA_MAVEN_TARGET"

bootstrap.sh (Root-Level)

#!/usr/bin/env bash
# bootstrap.sh (L1-133) -- key sections shown

# Mode selection: full | java | default | wrapper
MODE=$1  # or "default" if no argument

# Maven wrapper setup
MAVEN_CMD=$(get_env_var_with_optional_default VESPA_MAVEN_COMMAND "$(pwd)/mvnw")
mvn -B wrapper:wrapper -Dmaven=3.9.12 -N ${MAVEN_EXTRA_OPTS}

# Proxy script allowing plain "mvn" to use the wrapper
wbdir=maven-wrapper/bin
mkdir -p ${wbdir}
printf '#!/bin/sh\nexec %s/mvnw "$@"\n' "$(pwd)" > ${wbdir}/mvn
chmod +x ${wbdir}/mvn

# Core build function
mvn_install() {
    ${MAVEN_CMD} --batch-mode --no-snapshot-updates \
        -Dmaven.wagon.http.retryHandler.count=5 \
        clean "${MAVEN_TARGET}" ${MAVEN_EXTRA_OPTS} "$@"
}

# Build order:
# 1. Parent POMs (dependency-versions, container-dependency-versions, parent)
# 2. Root POM (non-recursive)
# 3. Maven plugins
# 4. Mode-specific modules (full mode: jrt, linguistics, messagebus)

cd dependency-versions && mvn_install
cd container-dependency-versions && mvn_install
cd parent && mvn_install
mvn_install -N
mvn_install -f maven-plugins/pom.xml

# Full mode: build C++ test dependencies
mvn_install -am -T1C -Dmaven.test.skip=true \
    -Dmaven.javadoc.skip=true -Dmaven.source.skip=true \
    -pl jrt,linguistics,messagebus

I/O Contract

Inputs (Environment Variables)

Variable	Required	Script	Description	Example
SOURCE_DIR	Yes	bootstrap.sh, java.sh	Root directory of the Vespa source checkout	/vespa
VESPA_CPP_TEST_JARS	Yes	bootstrap.sh	Destination directory for collected JAR files	/tmp/vespa-build/test-jars
NUM_MVN_THREADS	Yes	java.sh	Number of parallel Maven threads	4
VESPA_MAVEN_TARGET	Yes	java.sh, bootstrap.sh	Maven lifecycle target	install
VESPA_MAVEN_EXTRA_OPTS	Yes	java.sh, bootstrap.sh	Additional Maven options	-Dmaven.test.skip=true
VESPA_MAVEN_COMMAND	No	bootstrap.sh	Override for Maven command path	./mvnw
MAVEN_OPTS	No	java.sh	JVM arguments for Maven process	-Xms256m -Xmx2g
DEBUG	No	All	Enables bash xtrace when non-empty	1

Outputs (Files and Artifacts)

Output	Produced By	Description
./mvnw	bootstrap.sh (root)	Maven wrapper script (Maven 3.9.12)
maven-wrapper/bin/mvn	bootstrap.sh (root)	Proxy script redirecting mvn to the wrapper
~/.m2/repository/	bootstrap.sh (root), java.sh	Local Maven repository containing all compiled JARs
$VESPA_CPP_TEST_JARS/*.jar	.buildkite/bootstrap.sh	Flat directory of JAR files needed by C++ tests
dist/vtag.map	bootstrap.sh (root)	Version tag map generated by getversionmap.sh

Key Implementation Details

Maven Wrapper Setup

The root bootstrap.sh installs the Maven wrapper at a pinned version (3.9.12):

mvn -B wrapper:wrapper -Dmaven=3.9.12 -N ${MAVEN_EXTRA_OPTS}

The -N (non-recursive) flag ensures only the root module is processed. After installation, a proxy script is created at maven-wrapper/bin/mvn so that any script invoking plain mvn transparently uses the wrapper instead.

Build Order in bootstrap.sh

The root bootstrap script enforces a specific build order due to Maven's plugin resolution limitation:

1. dependency-versions -- Defines version constraints for third-party dependencies.
2. container-dependency-versions -- Defines version constraints for container-related dependencies.
3. parent -- The parent POM that all modules inherit from.
4. Root POM (non-recursive) -- Installs the root aggregator POM.
5. maven-plugins -- Custom Maven plugins used by other modules.
6. Mode-specific modules -- In full mode, builds jrt, linguistics, and messagebus with -am (also-make-dependencies) and -T1C (one thread per core).

JAR Collection for C++ Tests

The .buildkite/bootstrap.sh script uses a nested find command to locate all JAR files:

find . -type d -name target -exec find {} -mindepth 1 -maxdepth 1 -name "*.jar" \; \
    | xargs -I '{}' cp '{}' "$VESPA_CPP_TEST_JARS"

This two-level find first locates all Maven target/ directories, then within each one finds JAR files at the top level (not in subdirectories). The JARs are copied into a flat directory for easy classpath construction by C++ test binaries.

GCC Toolset and PATH Setup

The .buildkite/java.sh script activates the GCC toolset and adds Vespa dependency binaries to the PATH before invoking Maven:

source /etc/profile.d/enable-gcc-toolset.sh
PATH=/opt/vespa-deps/bin:$PATH

This is necessary because some Java modules contain JNI (Java Native Interface) code that requires native compilation during the Maven build.

Resource Limit Reporting

Before starting the build, java.sh optionally executes show-limits.sh to log system resource limits (e.g., open file descriptors, stack size). This diagnostic output helps troubleshoot build failures caused by resource exhaustion.

Execution Context

The typical invocation sequence in the Buildkite pipeline is:

Buildkite Pipeline
  --> .buildkite/bootstrap.sh
        --> cd $SOURCE_DIR && ./bootstrap.sh full
              --> mvn wrapper:wrapper (Maven 3.9.12)
              --> mvn_install dependency-versions
              --> mvn_install container-dependency-versions
              --> mvn_install parent
              --> mvn_install -N (root POM)
              --> mvn_install maven-plugins
              --> mvn_install jrt,linguistics,messagebus
        --> mkdir -p $VESPA_CPP_TEST_JARS
        --> find + xargs cp (collect JARs)
  --> .buildkite/java.sh
        --> source enable-gcc-toolset.sh
        --> ./mvnw -T $NUM_MVN_THREADS $VESPA_MAVEN_EXTRA_OPTS $VESPA_MAVEN_TARGET

Source File Locations

• .buildkite/bootstrap.sh (Lines 1-24)
• .buildkite/java.sh (Lines 1-31)
• bootstrap.sh (Lines 1-133)

See Also

• Java Bootstrap and Maven Build Principle -- The design rationale for the two-phase Java build.
• Prepare.sh Implementation -- The preceding pipeline stage.
• CMake Configuration Implementation -- The next stage that depends on bootstrap output.

Related Pages

• Principle:Vespa_engine_Vespa_Java_Bootstrap_and_Maven_Build
• Environment:Vespa_engine_Vespa_Java_17_Build_Runtime
• Heuristic:Vespa_engine_Vespa_Maven_Parallel_Build_Optimization