Implementation:Vespa engine Vespa Publish Artifacts Sh

Overview

This page documents the implementation of the Vespa artifact signing and publishing script: .buildkite/publish-artifacts.sh. This script creates tar archives of the RPM and Maven repositories, signs each artifact using Sigstore cosign with Buildkite OIDC identity, and uploads everything to the configured artifact destination.

Type: External Tool Doc

Code Reference

#!/usr/bin/env bash
# .buildkite/publish-artifacts.sh (L1-31)

set -o errexit
set -o nounset
set -o pipefail

if [[ -n "${DEBUG:-}" ]]; then
    set -o xtrace
fi

echo "--- Publishing build artifacts"
cd "$WORKDIR/artifacts/$ARCH"

echo "Creating archives..."
tar -cf rpm-repo.tar rpms &
tar -cf maven-repo.tar maven-repo
cp -a rpms/vespa-config-model-fat-*.rpm .
wait

echo "Signing artifacts..."
for FILE in *.tar *.rpm; do
    cosign sign-blob -y --oidc-provider=buildkite-agent \
        --output-signature "$FILE.sig" \
        --output-certificate "$FILE.pem" "$FILE"
done

ARTIFACT_DESTINATION="${VESPA_ENGINE_ARTIFACTS_BUCKET}/${VESPA_ENGINE_ARTIFACTS_PREFIX}/${VESPA_VERSION}/artifacts/${ARCH}"
echo "Uploading artifacts to ${ARTIFACT_DESTINATION} ..."
buildkite-agent artifact upload "*.tar;*.tar.sig;*.tar.pem;*.rpm;*.rpm.sig;*.rpm.pem" \
    "$ARTIFACT_DESTINATION"

I/O Contract

Inputs (Environment Variables)

Variable	Required	Description	Example
`WORKDIR`	Yes	Working directory containing build artifacts	`/tmp/vespa-build`
`ARCH`	Yes	CPU architecture identifier	`x86_64` or `aarch64`
`VESPA_ENGINE_ARTIFACTS_BUCKET`	Yes	Cloud storage bucket for uploads	`s3://vespa-artifacts`
`VESPA_ENGINE_ARTIFACTS_PREFIX`	Yes	Path prefix within the bucket	`builds`
`VESPA_VERSION`	Yes	Version string for the artifact path	`8.432.17`
`DEBUG`	No	If set to a non-empty value, enables bash xtrace	`1`

Inputs (Prerequisite Files)

File/Directory	Produced By	Description
`$WORKDIR/artifacts/$ARCH/rpms/`	build-rpms.sh	YUM repository with RPM packages and metadata
`$WORKDIR/artifacts/$ARCH/maven-repo/`	java.sh	Maven local repository with compiled JARs

Inputs (System Dependencies)

Tool	Purpose	Authentication
`cosign`	Sigstore keyless signing	Buildkite OIDC token (automatic)
`buildkite-agent`	Artifact upload to Buildkite	Agent token (pre-configured)
`tar`	Archive creation	N/A

Outputs (Files Created Locally)

Output	Type	Description
`rpm-repo.tar`	Tar archive	Archive of the complete YUM repository
`maven-repo.tar`	Tar archive	Archive of the Maven local repository
`vespa-config-model-fat-*.rpm`	RPM file	Extracted fat config model RPM
`*.sig` files	Detached signatures	Cosign signatures for each artifact
`*.pem` files	Certificates	Sigstore Fulcio certificates for each artifact

Outputs (Uploaded Artifacts)

All files matching *.tar, *.tar.sig, *.tar.pem, *.rpm, *.rpm.sig, and *.rpm.pem are uploaded to:

${VESPA_ENGINE_ARTIFACTS_BUCKET}/${VESPA_ENGINE_ARTIFACTS_PREFIX}/${VESPA_VERSION}/artifacts/${ARCH}/

Key Implementation Details

Parallel Archive Creation

tar -cf rpm-repo.tar rpms &
tar -cf maven-repo.tar maven-repo
cp -a rpms/vespa-config-model-fat-*.rpm .
wait

The RPM repository archive is created in a background process (&), while the Maven repository archive runs in the foreground. The cp -a command extracts the fat config model RPM for separate distribution. The wait command synchronizes with the background process before proceeding.

This parallel approach reduces total archiving time. Both tar operations are I/O-bound, so they can share disk bandwidth. The archives are uncompressed (-cf without a compression flag) because the downstream upload and any further packaging will handle compression.

Signing Loop

for FILE in *.tar *.rpm; do
    cosign sign-blob -y --oidc-provider=buildkite-agent \
        --output-signature "$FILE.sig" \
        --output-certificate "$FILE.pem" "$FILE"
done

The loop iterates over all tar archives and RPM files in the working directory. For each file, cosign performs keyless signing:

-y: Automatically accepts the Sigstore terms of service (non-interactive mode for CI).
--oidc-provider=buildkite-agent: Uses the Buildkite agent's built-in OIDC provider to obtain an identity token. The token contains claims identifying the Buildkite organization, pipeline, and build.
--output-signature "$FILE.sig": Writes the detached signature to a .sig file alongside the artifact.
--output-certificate "$FILE.pem": Writes the short-lived Fulcio certificate to a .pem file.

Each signing operation involves network calls to:

The Buildkite OIDC endpoint (for identity token)
Sigstore Fulcio (for certificate issuance)
Sigstore Rekor (for transparency log recording)

Artifact Upload

ARTIFACT_DESTINATION="${VESPA_ENGINE_ARTIFACTS_BUCKET}/${VESPA_ENGINE_ARTIFACTS_PREFIX}/${VESPA_VERSION}/artifacts/${ARCH}"
buildkite-agent artifact upload "*.tar;*.tar.sig;*.tar.pem;*.rpm;*.rpm.sig;*.rpm.pem" "$ARTIFACT_DESTINATION"

The buildkite-agent artifact upload command uploads matching files to the specified destination. Key aspects:

The glob patterns are semicolon-separated, matching multiple file types in a single command.
The destination path is hierarchically structured: bucket / prefix / version / artifacts / architecture.
The Buildkite agent handles authentication to the underlying storage backend (S3, GCS, etc.) using pre-configured credentials.

Working Directory Management

cd "$WORKDIR/artifacts/$ARCH"

The script changes to the architecture-specific artifact directory at the outset. All subsequent operations (archiving, signing, uploading) use relative paths within this directory. This simplifies the tar and glob commands and ensures the archive internal paths are clean (e.g., rpms/vespa-8.432.17-1.x86_64.rpm rather than /tmp/vespa-build/artifacts/x86_64/rpms/...).

Error Handling

The script uses strict error handling (set -o errexit, set -o nounset, set -o pipefail). If any command fails -- archiving, signing, or uploading -- the script terminates immediately. This is important because:

A failed signing operation should not be followed by an upload of unsigned artifacts.
A failed upload should be clearly reported rather than silently skipped.
The wait command propagates the exit code of the background tar process.

Execution Context

Buildkite Pipeline
  --> .buildkite/build-rpms.sh (produces RPMs)
  --> .buildkite/java.sh (produces Maven repo)
  --> .buildkite/publish-artifacts.sh
        --> cd $WORKDIR/artifacts/$ARCH
        --> tar -cf rpm-repo.tar rpms &
        --> tar -cf maven-repo.tar maven-repo
        --> cp vespa-config-model-fat RPM
        --> wait (synchronize background tar)
        --> cosign sign-blob (for each *.tar and *.rpm)
        --> buildkite-agent artifact upload

Source File Locations

.buildkite/publish-artifacts.sh (Lines 1-31)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment