Principle:Duckdb Duckdb Release Artifact Upload
Overview
Distributing release artifacts across multiple channels including GitHub Releases, S3 staging, and package registries. Release artifact upload is the final step of the DuckDB release pipeline, where validated build artifacts are published to the channels from which end users and downstream build systems consume them.
Description
DuckDB's release process distributes artifacts through three distinct channels, each serving a different audience and purpose:
Channel 1: GitHub Releases
GitHub Releases serve as the primary public-facing distribution channel. Artifacts are attached to tagged releases using the gh CLI tool. Typical artifacts include:
libduckdb-src.zip-- amalgamated source packageduckdb_cli-linux-amd64.zip-- pre-compiled CLI binaries for Linuxduckdb_cli-osx-universal.zip-- pre-compiled CLI binaries for macOSlibduckdb-linux-amd64.zip-- pre-compiled shared library for Linux- Platform-specific packages for Windows, ARM, and other architectures
Channel 2: S3 Staging
S3 staging buckets serve as an intermediate distribution layer used by CI and automated build systems. Artifacts are organized by commit hash, git describe version, repository name, and folder:
s3://duckdb-staging/<commit_hash>/<git_describe>/<repo_name>/<folder>/
This structure allows CI systems to fetch specific builds by commit or version without relying on GitHub Releases. It also provides a staging area where artifacts can be validated before being promoted to public release.
Channel 3: Package Registries (PyPI)
Python wheels are uploaded to the Python Package Index (PyPI) using twine. This enables end users to install DuckDB via:
pip install duckdb
PyPI uploads are immutable -- once a version is published, it cannot be overwritten. This requires careful version management to avoid conflicts.
Artifact Immutability
A core principle of release artifact management is immutability: once an artifact is uploaded to a distribution channel, it should not be modified or replaced (except in extraordinary circumstances). This ensures that:
- Downstream consumers always get the same artifact for a given version
- Build reproducibility is maintained
- Security auditing is possible (artifacts can be verified against known checksums)
Usage
This principle applies when:
- After package validation -- only validated packages should be uploaded
- As the final step of the release pipeline -- upload follows generation, amalgamation, packaging, and validation
- During CI-triggered releases -- automated workflows upload artifacts when a version tag is pushed
- For nightly or development builds -- artifacts are uploaded to S3 staging for testing
Theoretical Basis
| Concept | Description |
|---|---|
| Multi-Channel Distribution | Publishing artifacts through multiple channels (GitHub, S3, PyPI) to reach different audiences: end users (GitHub, PyPI), CI systems (S3), and package managers (PyPI). |
| Immutable Release Artifacts | Once published, artifacts are not modified. This ensures reproducibility and auditability. Version numbers are never reused. |
| Package Registry Publishing | Uploading to standardized package registries (PyPI, npm, etc.) that provide dependency resolution, version management, and platform-specific package selection for end users. |