Implementation:Duckdb Duckdb Run All Generators

Overview

Concrete tool for executing all DuckDB code generation scripts as a prerequisite for source amalgamation. This is a composite step that sequentially invokes every scripts/generate_*.py script, ensuring all generated C and C++ source files are current before amalgamation or packaging proceeds.

Code Reference

The following scripts comprise the full generation suite. They must all be executed (in any order, though the listed order is conventional):

Script	Source Location	Primary Output
`generate_c_api.py`	`scripts/generate_c_api.py`	`src/include/duckdb.h` (public C API header)
`generate_enum.py`	`scripts/generate_enum.py`	Enum class definitions in `src/include/duckdb/common/enums/`
`generate_serialization.py`	`scripts/generate_serialization.py`	Serialization/deserialization routines in `src/`
`generate_grammar.py`	`scripts/generate_grammar.py`	Parser grammar files (bison/flex output in `third_party/libpg_query/`)
`generate_functions.py`	`scripts/generate_functions.py`	Function registration code in `src/`
`generate_settings.py`	`scripts/generate_settings.py`	Settings registration in `src/`
`generate_metrics.py`	`scripts/generate_metrics.py`	Profiling metrics definitions in `src/`

I/O Contract

External Dependencies

Dependency	Version	Purpose
`python3`	3.7+	Runs all generation scripts
`bison`	3.0+	Required by `generate_grammar.py` for parser generation
`flex`	2.6+	Required by `generate_grammar.py` for lexer generation

Inputs

JSON specification files -- enum definitions, function specs, serialization descriptors (various locations under src/)
Grammar fragments -- SQL grammar rules in third_party/libpg_query/grammar/
Enum headers -- existing enum header files used as templates
Template files -- code templates that generators expand into full source

Outputs

All generated C/C++ source files needed before amalgamation, including:
- Public C API header (src/include/duckdb.h)
- Enum class headers
- Serialization routines
- Parser grammar (.cpp / .hpp from bison/flex)
- Function registration code
- Settings registration code
- Metrics definitions

Exit Behavior

Each script exits with code 0 on success and non-zero on failure. The composite step must abort if any individual script fails, since downstream amalgamation would produce incomplete output.

Usage Examples

Running All Generators Sequentially (Shell)

#!/usr/bin/env bash
set -euo pipefail

cd /path/to/duckdb

echo "Running all code generators..."

python3 scripts/generate_c_api.py
echo "[1/7] generate_c_api.py complete"

python3 scripts/generate_enum.py
echo "[2/7] generate_enum.py complete"

python3 scripts/generate_serialization.py
echo "[3/7] generate_serialization.py complete"

python3 scripts/generate_grammar.py
echo "[4/7] generate_grammar.py complete"

python3 scripts/generate_functions.py
echo "[5/7] generate_functions.py complete"

python3 scripts/generate_settings.py
echo "[6/7] generate_settings.py complete"

python3 scripts/generate_metrics.py
echo "[7/7] generate_metrics.py complete"

echo "All generators finished successfully."

Running Generators Then Amalgamation

# Full pipeline: generate -> amalgamate -> package
python3 scripts/generate_c_api.py && \
python3 scripts/generate_enum.py && \
python3 scripts/generate_serialization.py && \
python3 scripts/generate_grammar.py && \
python3 scripts/generate_functions.py && \
python3 scripts/generate_settings.py && \
python3 scripts/generate_metrics.py && \
python3 scripts/amalgamation.py && \
python3 scripts/package_build.py

Running a Single Generator (Development)

# After modifying a JSON enum spec, re-run only the enum generator:
python3 scripts/generate_enum.py

# After modifying grammar rules, re-run only the grammar generator:
python3 scripts/generate_grammar.py

Makefile Integration

.PHONY: generate amalgamation

generate:
	python3 scripts/generate_c_api.py
	python3 scripts/generate_enum.py
	python3 scripts/generate_serialization.py
	python3 scripts/generate_grammar.py
	python3 scripts/generate_functions.py
	python3 scripts/generate_settings.py
	python3 scripts/generate_metrics.py

amalgamation: generate
	python3 scripts/amalgamation.py

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment