Environment:Duckdb Duckdb Extension Distribution Env
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, Distribution |
| Last Updated | 2026-02-07 12:00 GMT |
Overview
CI/CD environment with AWS CLI, OpenSSL, gzip, brotli, and truncate for signing, compressing, and uploading DuckDB extensions to S3.
Description
This environment provides the tools required for the extension distribution pipeline: signing extension binaries with RSA keys via OpenSSL, compressing them with gzip (or brotli for WASM), and uploading to S3 buckets using the AWS CLI. The pipeline includes a dry-run safety mode controlled by the `DUCKDB_DEPLOY_SCRIPT_MODE` environment variable. Extensions are uploaded to both versioned and latest paths in S3 for distribution.
Usage
Use this environment when running the extension signing, uploading, and promotion scripts. This is required for the Extension_Development_And_Distribution workflow steps involving binary signing, S3 upload, upload testing, and nightly-to-production promotion.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| OS | Linux or macOS | Shell scripts use bash |
| Hardware | Any | No special hardware requirements |
| Network | S3 access | Requires connectivity to AWS S3 endpoints |
Dependencies
System Packages
- `openssl` (for RSA signature generation via `pkeyutl`)
- `gzip` (for extension compression on native platforms)
- `brotli` (for extension compression on WASM platforms)
- `aws` CLI (for S3 upload operations)
- `truncate` or `gtruncate` (for removing trailing metadata bytes)
- `sha256sum` or equivalent (for hash computation)
- `bash` (scripts use bash-specific features)
Credentials
The following environment variables must be set for deployment:
- `AWS_ACCESS_KEY_ID`: AWS access key for S3 upload operations
- `AWS_SECRET_ACCESS_KEY`: AWS secret key for S3 upload operations
- `DUCKDB_EXTENSION_SIGNING_PK`: RSA private key (PEM format) for extension signing (optional; defaults to 256 zero bytes if not set)
- `DUCKDB_DEPLOY_SCRIPT_MODE`: Set to `for_real` to disable dry-run mode (safety guard)
Quick Install
# Ubuntu/Debian
sudo apt-get install -y openssl gzip brotli coreutils
pip install awscli
# macOS
brew install openssl brotli coreutils awscli
# Verify tools
openssl version
aws --version
gzip --version
Code Evidence
AWS key check and dry-run safety from `scripts/extension-upload-single.sh:63-74`:
# Abort if AWS key is not set
if [ -z "$AWS_ACCESS_KEY_ID" ]; then
echo "No AWS key found, skipping.."
rm "$ext.compressed"
exit 0
fi
# Set dry run unless guard var is set
DRY_RUN_PARAM="--dryrun"
if [ "$DUCKDB_DEPLOY_SCRIPT_MODE" == "for_real" ]; then
DRY_RUN_PARAM=""
fi
Extension signing with OpenSSL from `scripts/extension-upload-single.sh:39-47`:
if [ "$DUCKDB_EXTENSION_SIGNING_PK" != "" ]; then
echo "$DUCKDB_EXTENSION_SIGNING_PK" > private.pem
$script_dir/compute-extension-hash.sh $ext.append > $ext.hash
openssl pkeyutl -sign -in $ext.hash -inkey private.pem -pkeyopt digest:sha256 -out $ext.sign
rm -f private.pem
else
# Default to 256 zeros
dd if=/dev/zero of=$ext.sign bs=256 count=1
fi
Compression with gzip/brotli from `scripts/extension-upload-single.sh:53-58`:
if [[ $4 == wasm_* ]]; then
brotli < $ext.append > "$ext.compressed"
else
gzip < $ext.append > "$ext.compressed"
fi
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `No AWS key found, skipping..` | `AWS_ACCESS_KEY_ID` not set | Export AWS credentials before running upload scripts |
| `openssl: command not found` | OpenSSL not installed | Install openssl: `sudo apt-get install openssl` |
| Dry run output (no actual upload) | `DUCKDB_DEPLOY_SCRIPT_MODE` not set to `for_real` | Set `DUCKDB_DEPLOY_SCRIPT_MODE=for_real` for actual uploads |
| `truncate: command not found` | coreutils not installed (macOS) | Install via `brew install coreutils` (provides `gtruncate`) |
Compatibility Notes
- macOS: Uses `gtruncate` instead of `truncate` (auto-detected in script).
- WASM extensions: Use brotli compression instead of gzip, and set `content-encoding: br` and `content-type: application/wasm` headers on S3.
- Dry-run mode: All uploads default to `--dryrun` mode. Must explicitly set `DUCKDB_DEPLOY_SCRIPT_MODE=for_real` to perform actual uploads.
- S3 buckets: Nightly builds go to `duckdb-extensions-nightly`, production to `duckdb-extensions`.