Principle:Duckdb Duckdb Declarative Benchmark Execution

Overview

Declarative Benchmark Execution is the principle of defining and executing performance benchmarks using a declarative domain-specific language (DSL) rather than imperative code. Instead of writing C++ benchmark classes for every SQL-based performance test, benchmarks are specified in plain-text .benchmark files that declare what to run, what result to expect, and how to initialize the environment. The framework's interpreter translates these declarations into the standard benchmark lifecycle (initialize, run, verify, cleanup) without the benchmark author writing any procedural code.

Description

The central idea of Declarative Benchmark Execution is the separation of benchmark definition from benchmark execution. The benchmark author focuses on what to measure, while the framework handles how to measure it.

Key aspects of this principle include:

• DSL-based benchmark specification -- Benchmarks are written in a simple, line-oriented DSL. Each .benchmark file contains keyword-value pairs that define the benchmark's name, group, initialization SQL, the query to measure, and the expected result.
• Separation of definition from execution -- The benchmark file is purely declarative. It does not contain any control flow, loop constructs, or procedural logic. The benchmark runner's interpreter reads the file and executes each phase of the benchmark lifecycle according to the declared specification.
• The .benchmark file format -- Files use a keyword-driven format where each section begins with a keyword (e.g., name, group, load, run, result) followed by the content for that section. This format is easy to read, write, and diff.
• Standardized benchmark lifecycle -- Every declarative benchmark follows the same lifecycle:
  1. Init -- Execute the load SQL statements to set up the database schema and data.
  2. Run -- Execute the run SQL statement, which is the query being benchmarked.
  3. Verify -- Compare the query output against the declared result to ensure correctness.
  4. Cleanup -- Tear down state as needed between runs.
• Template and parameterization support -- The DSL supports templates and variable substitution, allowing a single benchmark definition to be parameterized across different configurations (e.g., different scale factors or data types).

Usage

Apply Declarative Benchmark Execution when:

• SQL-based benchmarks need to be defined and run with expected result verification.
• Benchmark authors are SQL experts who should not need to write C++ code.
• A large number of benchmarks need to be maintained, and a concise, uniform format reduces maintenance cost.
• Benchmark definitions need to be easily reviewable in code review (plain-text, diffable format).
• The benchmark lifecycle is standardized and does not require custom procedural logic.

Theoretical Basis

This principle draws on several foundational concepts:

• Declarative specification -- A programming paradigm where the user describes the desired outcome rather than the steps to achieve it. SQL itself is a declarative language; the .benchmark DSL extends this philosophy to benchmark definition.
• Benchmark lifecycle (init, run, verify, cleanup) -- A standardized four-phase model for benchmark execution, analogous to the setup/exercise/verify/teardown pattern in unit testing (the "Four-Phase Test" pattern).
• Domain-specific languages -- Small, focused languages designed for a specific problem domain. The .benchmark DSL is a domain-specific language for performance benchmark specification.
• Separation of concerns -- The benchmark definition (what to test) is cleanly separated from the benchmark infrastructure (how to test it). Changes to the runner do not require changes to benchmark files, and vice versa.

Related

• Implementation:Duckdb_Duckdb_Interpreted_Benchmark