Environment:Duckdb Duckdb Release Publishing Env
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, Release_Management |
| Last Updated | 2026-02-07 12:00 GMT |
Overview
CI/CD environment with GitHub CLI (gh), AWS CLI, and Python packaging tools for publishing DuckDB release artifacts to GitHub Releases, S3 staging, and PyPI.
Description
This environment supports the final stage of the DuckDB release pipeline: uploading compiled artifacts to their distribution endpoints. The `asset-upload-gha.py` script uses the GitHub CLI to upload assets to GitHub Releases during tag builds. The `upload-assets-to-staging.sh` script uploads to an S3 staging bucket. The `release-pip.py` script downloads pre-built wheel packages and publishes them to PyPI. All scripts include safety guards to prevent accidental uploads from wrong repositories or branches.
Usage
Use this environment when publishing release artifacts. Required for the Release_Artifact_Upload implementation in the Source_Amalgamation_And_Packaging workflow. Only activated during tagged releases in the `duckdb/duckdb` repository.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| OS | Linux or macOS | CI runners (GitHub Actions) |
| Hardware | Any | No special hardware requirements |
| Network | Internet access | Requires connectivity to GitHub, S3, and PyPI |
Dependencies
System Packages
- `gh` (GitHub CLI) for uploading release assets
- `aws` CLI for S3 staging uploads
- `bash` for shell scripts
Python Packages
- `github3.py` (for GitHub API interactions in asset-upload scripts)
- `twine` (for PyPI uploads in release-pip.py)
- `pip` (for downloading wheel packages)
Credentials
The following environment variables must be set:
- `GH_TOKEN` or `GITHUB_TOKEN`: GitHub token for `gh release upload` operations
- `AWS_ACCESS_KEY_ID`: AWS access key for S3 staging uploads
- `AWS_SECRET_ACCESS_KEY`: AWS secret key for S3 staging uploads
- `TWINE_USERNAME`: PyPI username (for pip release)
- `TWINE_PASSWORD`: PyPI password/token (for pip release)
- `DUCKDB_DEPLOY_SCRIPT_MODE`: Set to `for_real` to disable dry-run safety mode
Quick Install
# Install GitHub CLI
sudo apt-get install -y gh
# Install Python packages
pip install github3.py twine
# Install AWS CLI
pip install awscli
Code Evidence
Repository safety guard from `scripts/asset-upload-gha.py:28-35` (approximate):
# Only upload from the main duckdb/duckdb repository
repo = os.environ.get('GITHUB_REPOSITORY', '')
if repo != 'duckdb/duckdb':
print(f"Not uploading from {repo}")
exit(0)
S3 staging safety guard from `scripts/upload-assets-to-staging.sh:15-20` (approximate):
# Abort if AWS key is not set
if [ -z "$AWS_ACCESS_KEY_ID" ]; then
echo "No AWS key found, skipping.."
exit 0
fi
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `Not uploading from <repo>` | Running from a fork, not duckdb/duckdb | Only runs from the official repository during tag builds |
| `gh: command not found` | GitHub CLI not installed | Install via `sudo apt-get install gh` or `brew install gh` |
| `No AWS key found, skipping..` | AWS credentials not configured | Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` |
Compatibility Notes
- GitHub Actions only: The `asset-upload-gha.py` script is designed specifically for GitHub Actions CI environment.
- Tag builds only: Release uploads only trigger on tagged commits matching the version pattern.
- Safety guards: All upload scripts default to dry-run mode and check repository identity to prevent accidental uploads from forks.