Implementation:Duckdb Duckdb Package Build Py

Overview

Concrete tool for building distributable DuckDB source and binary packages. The package_build.py script assembles amalgamated source files, public headers, third-party dependencies, and extension source into zip archives ready for distribution via GitHub Releases and other channels.

Code Reference

Source Location: scripts/package_build.py (lines 1--454)

Key Functions

Function	Signature	Purpose
`third_party_includes`	`third_party_includes()`	Returns a list of all third-party include directories needed for compilation. Scans CMakeLists.txt files under `third_party/`.
`third_party_sources`	`third_party_sources()`	Returns a list of all third-party source files that must be compiled alongside the amalgamated source.
`get_libraries`	`get_libraries(binary_dir, libraries, extensions)`	Locates static and shared library files in a build directory. Used when assembling binary (pre-compiled) packages.
`includes`	`includes(extensions)`	Computes the full set of include paths needed to compile DuckDB, including core includes, third-party includes, and extension includes.
`get_git_describe`	`get_git_describe()`	Runs `git describe --tags --long` to compute the version string from the most recent git tag. Returns strings like `v0.9.2-0-gabcdef1`.
`git_commit_hash`	`git_commit_hash()`	Returns the full SHA-1 hash of the current git HEAD commit.
`git_dev_version`	`git_dev_version()`	Computes the development version string used by CI. Typically appends a commit count and short hash to the base version.

I/O Contract

Command-Line Interface

python3 scripts/package_build.py [OPTIONS]

Options:
  --no-git              Skip git-based version detection (use fallback version)
  --extensions ext1;ext2  Semicolon-delimited list of extensions to include

External Dependencies

Dependency	Purpose
`python3` (3.7+)	Script runtime
`git`	Version detection via `git describe` and `git rev-parse`
`zip`	Archive creation for source packages

Inputs

Amalgamated source files: src/amalgamation/duckdb.cpp and src/amalgamation/duckdb.hpp (produced by amalgamation.py)
Public headers: src/include/duckdb.h (C API), src/include/duckdb_extension.h (Extension API)
Extension source files: source and headers for any extensions specified via --extensions
CMakeLists.txt files: used to discover third-party source and include directories
Git repository state: tags and commit history for version computation

Outputs

Output	Contents
`libduckdb-src.zip`	Self-contained source package: `duckdb.hpp`, `duckdb.cpp`, `duckdb.h`, `duckdb_extension.h`, plus any third-party sources needed for compilation
Platform binary packages	Pre-compiled libraries and headers for specific platforms (e.g., `libduckdb-linux-amd64.zip`)

Usage Examples

Basic Source Package

# First, run amalgamation (prerequisite)
python3 scripts/amalgamation.py

# Then build the source package
python3 scripts/package_build.py
# Produces: libduckdb-src.zip

Source Package with Extensions

# Include parquet and httpfs extensions in the package
python3 scripts/package_build.py --extensions "parquet;httpfs"

Package Without Git (Offline Build)

# Build package without git version detection (uses fallback version)
python3 scripts/package_build.py --no-git

Full Release Pipeline

#!/usr/bin/env bash
set -euo pipefail

# Step 1: Run all generators
python3 scripts/generate_c_api.py
python3 scripts/generate_enum.py
python3 scripts/generate_serialization.py
python3 scripts/generate_grammar.py
python3 scripts/generate_functions.py
python3 scripts/generate_settings.py
python3 scripts/generate_metrics.py

# Step 2: Create amalgamation
python3 scripts/amalgamation.py --extended

# Step 3: Build source package
python3 scripts/package_build.py --extensions "parquet;httpfs;json"

# Step 4: Verify the package
unzip -l libduckdb-src.zip

Inspecting Version Detection

# In a Python session, test version detection:
import sys
sys.path.insert(0, 'scripts')
from package_build import get_git_describe, git_commit_hash, git_dev_version

print(f"Version:    {get_git_describe()}")
print(f"Commit:     {git_commit_hash()}")
print(f"Dev version: {git_dev_version()}")

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment