Implementation:Duckdb Duckdb Release Upload Scripts

Overview

Concrete tools for uploading DuckDB release artifacts to GitHub Releases, S3 staging, and PyPI. This implementation comprises three separate scripts, each targeting a specific distribution channel. Together, they form the final step of the DuckDB release pipeline.

Code Reference

Script 1: asset-upload-gha.py (GitHub Releases)

Source Location: scripts/asset-upload-gha.py (lines 1--126)

Purpose: Uploads build artifacts to a GitHub Release by invoking the gh release upload command. Designed to run inside GitHub Actions workflows.

Key Behavior: Iterates over the provided file arguments and attaches each to the current GitHub Release tag. Handles retries and error reporting for failed uploads.

Script 2: upload-assets-to-staging.sh (S3 Staging)

Source Location: scripts/upload-assets-to-staging.sh (lines 1--65)

Purpose: Uploads build artifacts to an S3 staging bucket organized by commit hash and version. Used by CI systems to make artifacts available before the formal GitHub Release is published.

Key Behavior: Computes the S3 path from git metadata (commit hash, git describe output, repository name) and uploads files using the aws s3 cp command.

Script 3: release-pip.py (PyPI)

Source Location: tools/release-pip.py (lines 1--53)

Purpose: Uploads Python wheel packages to the Python Package Index (PyPI) using twine upload. Enables pip install duckdb for end users.

Key Behavior: Locates wheel files in the build output directory and invokes twine upload with appropriate credentials (typically from environment variables or a .pypirc file).

I/O Contract

Command-Line Interfaces

asset-upload-gha.py

python3 scripts/asset-upload-gha.py <file1> [file2] [file3] ...

Arguments:
  <files...>   One or more artifact files to attach to the GitHub Release.
               The script uses `gh release upload` internally.

upload-assets-to-staging.sh

scripts/upload-assets-to-staging.sh <folder> <file1> [file2] [file3] ...

Arguments:
  <folder>     Subfolder name within the S3 staging path (e.g., "linux-amd64")
  <files...>   One or more artifact files to upload to S3.

Upload path:
  s3://duckdb-staging/<commit_hash>/<git_describe>/<repo_name>/<folder>/

release-pip.py

python3 tools/release-pip.py

No arguments required. The script discovers wheel files in the standard
build output locations and uploads them to PyPI.

External Dependencies

Dependency	Required By	Purpose
`gh` CLI	`asset-upload-gha.py`	GitHub CLI for interacting with GitHub Releases. Must be authenticated.
`aws` CLI	`upload-assets-to-staging.sh`	AWS CLI for S3 uploads. Must be configured with appropriate credentials.
`twine`	`release-pip.py`	Python package for uploading distributions to PyPI.
`pip`	`release-pip.py`	Used to install/manage twine if not already available.
`python3`	All scripts	Script runtime.
`git`	`upload-assets-to-staging.sh`	Used to compute commit hash and version for S3 path construction.

Inputs

Build artifacts produced by earlier pipeline stages:

Artifact	Description	Typical Source
`libduckdb-src.zip`	Amalgamated source package	`package_build.py`
`duckdb_cli-linux-amd64.zip`	Linux CLI binary	CMake build
`duckdb_cli-osx-universal.zip`	macOS CLI binary	CMake build
`libduckdb-linux-amd64.zip`	Linux shared library	CMake build
`duckdb-*.whl`	Python wheel packages	`setup.py bdist_wheel`

Outputs

Channel	Location	Access Method
GitHub Releases	`https://github.com/duckdb/duckdb/releases/tag/vX.Y.Z`	Browser, `gh release download`, direct URL
S3 Staging	`s3://duckdb-staging/<commit>/<version>/<repo>/<folder>/`	`aws s3 cp`, HTTP (if bucket is public)
PyPI	`https://pypi.org/project/duckdb/`	`pip install duckdb==X.Y.Z`

Usage Examples

Upload to GitHub Releases

# Upload source package and CLI binaries to the current release
python3 scripts/asset-upload-gha.py \
    libduckdb-src.zip \
    duckdb_cli-linux-amd64.zip \
    duckdb_cli-osx-universal.zip \
    libduckdb-linux-amd64.zip

Upload to S3 Staging

# Upload Linux artifacts to S3 staging
scripts/upload-assets-to-staging.sh linux-amd64 \
    libduckdb-linux-amd64.zip \
    duckdb_cli-linux-amd64.zip

# The files will be placed at:
# s3://duckdb-staging/<commit>/<git_describe>/duckdb/linux-amd64/

Upload Python Wheel to PyPI

# Ensure twine is installed
pip install twine

# Upload the wheel to PyPI
python3 tools/release-pip.py

# End users can then install with:
# pip install duckdb==0.9.2

Full Release Upload Pipeline

#!/usr/bin/env bash
set -euo pipefail

# Assumes all artifacts have been built and validated

echo "Uploading to S3 staging..."
scripts/upload-assets-to-staging.sh linux-amd64 \
    libduckdb-src.zip \
    libduckdb-linux-amd64.zip \
    duckdb_cli-linux-amd64.zip

echo "Uploading to GitHub Releases..."
python3 scripts/asset-upload-gha.py \
    libduckdb-src.zip \
    libduckdb-linux-amd64.zip \
    duckdb_cli-linux-amd64.zip

echo "Uploading Python wheel to PyPI..."
python3 tools/release-pip.py

echo "All uploads complete."

GitHub Actions Integration

# Typical GitHub Actions step for artifact upload
- name: Upload release artifacts
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    python3 scripts/asset-upload-gha.py \
      libduckdb-src.zip \
      duckdb_cli-linux-amd64.zip

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment