Implementation:Apache Spark Do Release Docker
| Knowledge Sources | |
|---|---|
| Domains | Release_Engineering |
| Type | API Doc |
| Last Updated | 2026-02-08 12:00 GMT |
Overview
Dockerized release orchestrator that runs the Apache Spark release process inside a controlled container environment.
Description
do-release-docker.sh builds a Docker image with all release tools (Maven, GPG, Git, SBT) and runs the multi-step release process inside it. Supports single-step execution (-s flag), dry-run mode (-n flag), and custom JDK paths. Environment variables for version, credentials, and signing keys are injected via Docker env files.
The script serves as the primary entry point for all Apache Spark releases. It handles:
- Docker image construction: Builds a Docker image from the release Dockerfile containing all required tooling.
- Credential injection: Reads environment variables for ASF credentials, GPG keys, and version information, passing them securely to the container via Docker env files.
- Workspace mounting: Mounts the specified working directory into the container so artifacts persist after the container exits.
- Step orchestration: Delegates to
do-release.shinside the container, which in turn calls step-specific scripts (tag, build, docs, publish, finalize).
The inner script do-release.sh coordinates the release steps and calls utility functions from release-util.sh for common operations like version manipulation, git configuration, and Maven profile resolution.
Usage
Use as the primary entry point for creating Apache Spark releases. All release managers should use this script rather than running individual release steps manually.
Code Reference
Source Location
- Repository: apache/spark
- Files:
dev/create-release/do-release-docker.sh(lines 1-181)dev/create-release/do-release.sh(lines 1-91)dev/create-release/release-util.sh(lines 86-237)
Signature
dev/create-release/do-release-docker.sh -d <workdir> [-n] [-t <tag>] [-j <jdk-path>] [-s <step>]
Release Steps
The release process consists of five ordered steps:
| Step | Script | Description |
|---|---|---|
| tag | release-tag.sh |
Creates signed git tag and bumps version numbers |
| build | release-build.sh package |
Builds source and binary distribution tarballs |
| docs | release-build.sh docs |
Generates API documentation for all languages |
| publish | release-build.sh publish-release |
Uploads artifacts to Nexus staging and sends vote email |
| finalize | release-build.sh finalize |
Promotes artifacts after successful vote |
Key Environment Variables
| Variable | Description |
|---|---|
RELEASE_VERSION |
The version being released (e.g., 3.5.0)
|
RELEASE_TAG |
The git tag name (e.g., v3.5.0-rc1)
|
NEXT_VERSION |
The next development version (e.g., 3.5.1-SNAPSHOT)
|
GIT_BRANCH |
The release branch (e.g., branch-3.5)
|
ASF_USERNAME |
Apache Software Foundation username |
ASF_PASSWORD |
Apache Software Foundation password |
GPG_KEY |
GPG key identifier for signing |
GPG_PASSPHRASE |
Passphrase for the GPG signing key |
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
-d workdir |
directory path | Yes | Working directory for the release process |
-n |
flag | No | Enable dry-run mode (no publishing) |
-t tag |
string | No | Custom release tag override |
-j jdk-path |
directory path | No | Path to a custom JDK installation |
-s step |
string | No | Run a single step (tag, build, docs, publish, finalize) |
| Release config env vars | environment | Yes | Version, credential, and signing key variables |
| GPG signing key | GPG key | Yes | Release manager's GPG key for artifact signing |
| Docker environment | Docker daemon | Yes | Running Docker daemon accessible to the user |
Outputs
| Name | Type | Description |
|---|---|---|
| Docker container | Docker process | Container running the release process with all tools |
| Release artifacts | files | Source tarballs, binary distributions, signatures, and checksums in the working directory |
Usage Examples
Full Release
# Run the complete release process
./dev/create-release/do-release-docker.sh -d /tmp/spark-release
Dry Run
# Test the release process without publishing
./dev/create-release/do-release-docker.sh -d /tmp/spark-release -n
Single Step Execution
# Run only the build step
./dev/create-release/do-release-docker.sh -d /tmp/spark-release -s build
# Run only the tag step
./dev/create-release/do-release-docker.sh -d /tmp/spark-release -s tag
# Run only the publish step
./dev/create-release/do-release-docker.sh -d /tmp/spark-release -s publish