Principle:Apache Spark Release Artifact Publishing
| Domains | Release_Engineering, Distribution |
|---|---|
| Last Updated | 2026-02-08 12:00 GMT |
Overview
A staged artifact publishing process that uploads signed release candidates to staging repositories for community verification before final release.
Description
Open source releases follow a vote-based process. Artifacts are first uploaded to staging repositories (Apache Nexus for Maven, SVN for binaries) where the community can verify signatures and test the release candidate. Only after a successful vote are artifacts promoted to release repositories. This two-phase publishing prevents incomplete or faulty releases from reaching end users.
The Apache Spark publishing process involves multiple distribution channels:
- Apache Nexus: Maven artifacts (JARs, POMs, sources, javadocs) are uploaded to a Nexus staging repository. The staging repository is created, populated, and then closed, making the artifacts available for verification but not yet promoted to Maven Central.
- Apache SVN: Binary distribution tarballs, signatures, and checksums are uploaded to the Apache dev SVN repository for community verification.
- Mailing list: An RC vote email is sent to
dev@spark.apache.orgwith links to all artifacts, checksums, and verification instructions.
The staging approach provides a critical safety net:
- Reversibility: If the community vote fails, the staging repository can be dropped and the SVN artifacts removed without affecting end users.
- Verification window: Community members have time to download, verify signatures, run tests, and report issues before the release is finalized.
- Transparency: The vote email provides a single point of reference for all release artifacts and their verification.
Usage
Use after building all release artifacts and before calling the RC vote on the dev mailing list. This step makes artifacts available for the community to verify.
Theoretical Basis
The publishing process follows a staged release model:
create_staging_repo() -> upload(artifacts) -> close_staging_repo() -> call_vote(mailing_list) -> if_passed: promote(staging -> release)
The staged release model provides the following guarantees:
- Atomicity: The Nexus staging repository is either fully closed (all artifacts present) or dropped (nothing visible), preventing partial releases.
- Community governance: The vote-based process ensures that multiple community members have verified the release before it reaches end users.
- Audit trail: The mailing list vote thread provides a permanent record of who verified the release, what issues were found, and how they were resolved.
- Parallel verification: Multiple community members can simultaneously download and verify artifacts from the staging repository.
This process reflects the Apache Software Foundation's commitment to community-driven releases, where no single individual can unilaterally publish a release without community consensus.