Implementation:Duckdb Duckdb Benchmark Class

Overview

Benchmark_Class is the concrete tool for benchmark registration and discovery provided by DuckDB's benchmark framework. It defines the Benchmark abstract base class that all DuckDB benchmarks must inherit from, and the BenchmarkConfiguration struct that controls execution parameters. When a Benchmark subclass is instantiated with register_benchmark=true, it adds itself to a global registry, enabling the benchmark runner to discover and enumerate all available benchmarks at runtime.

Code Reference

Benchmark Base Class

Source: benchmark/include/benchmark.hpp:L30-93

class Benchmark {
    constexpr static size_t DEFAULT_NRUNS = 5;
public:
    string name;
    string group;

    Benchmark(bool register_benchmark, string name, string group);

    virtual unique_ptr<BenchmarkState> Initialize(BenchmarkConfiguration &config);
    virtual void Run(BenchmarkState *state) = 0;
    virtual void Cleanup(BenchmarkState *state) = 0;
    virtual string Verify(BenchmarkState *state) = 0;
    virtual void Interrupt(BenchmarkState *state) = 0;
    virtual string BenchmarkInfo() = 0;
    virtual string GetLogOutput(BenchmarkState *state) = 0;
    virtual bool RequireReinit();
    virtual size_t NRuns();
    virtual optional_idx Timeout(const BenchmarkConfiguration &config);
};

Key members:

• name -- Unique identifier for the benchmark (e.g., "tpch_q01").
• group -- Logical grouping (e.g., "tpch", "micro").
• Initialize() -- Sets up the benchmark state (database, loaded data).
• Run() -- Executes the benchmark workload (pure virtual).
• Cleanup() -- Resets state between runs (pure virtual).
• Verify() -- Checks correctness of results (pure virtual). Returns an empty string on success, or an error message on failure.
• Interrupt() -- Handles timeout interruption (pure virtual).
• NRuns() -- Returns the number of repetitions (default: 5).
• Timeout() -- Returns the per-run timeout duration.
• DEFAULT_NRUNS -- Compile-time constant defining the default number of runs (5).

BenchmarkConfiguration Struct

Source: benchmark/include/benchmark.hpp

struct BenchmarkConfiguration {
    constexpr static size_t DEFAULT_TIMEOUT = 30;

    string name_pattern {};
    BenchmarkMetaType meta = BenchmarkMetaType::NONE;
    BenchmarkProfileInfo profile_info = BenchmarkProfileInfo::NONE;
    optional_idx timeout_duration = optional_idx(DEFAULT_TIMEOUT);
};

Key members:

• name_pattern -- A regex or substring pattern used to filter which benchmarks are executed.
• meta -- Controls meta-information output (e.g., query plans).
• profile_info -- Controls profiling output mode.
• timeout_duration -- Maximum time in seconds for each benchmark run (default: 30 seconds).
• DEFAULT_TIMEOUT -- Compile-time constant for the default timeout (30 seconds).

I/O Contract

Inputs

• Compiled benchmark runner -- The benchmark_runner binary with all benchmark subclasses linked in. Each subclass that passes register_benchmark=true to the Benchmark constructor is added to the global registry at static initialization time.
• .benchmark files -- Declarative benchmark definition files (processed by the InterpretedBenchmark subclass, which also registers itself via this mechanism).

Outputs

• List of registered benchmark names -- When invoked with --list, the runner enumerates all registered benchmarks, displaying their name and group.
• Filtered benchmark set -- When a name pattern is provided, only benchmarks matching the pattern are selected for execution.

Usage Examples

Listing All Registered Benchmarks

# List all benchmarks discovered via the global registry
build/release/benchmark/benchmark_runner --list

Sample output:

benchmark/micro/aggregate/aggregate_few_groups.benchmark
benchmark/micro/aggregate/aggregate_many_groups.benchmark
benchmark/tpch/sf1/q01.benchmark
benchmark/tpch/sf1/q02.benchmark
...

Filtering by Name Pattern

# Run only benchmarks matching "tpch"
build/release/benchmark/benchmark_runner "tpch.*"

Subclassing Benchmark

To create a new C++ benchmark, subclass Benchmark and register it:

#include "benchmark.hpp"

class MyCustomBenchmark : public Benchmark {
public:
    MyCustomBenchmark() : Benchmark(true, "my_custom_benchmark", "custom") {}

    unique_ptr<BenchmarkState> Initialize(BenchmarkConfiguration &config) override {
        // Set up database and load data
        auto state = make_uniq<BenchmarkState>();
        // ... initialization logic ...
        return state;
    }

    void Run(BenchmarkState *state) override {
        // Execute the workload
    }

    void Cleanup(BenchmarkState *state) override {
        // Reset state between runs
    }

    string Verify(BenchmarkState *state) override {
        // Return empty string on success, error message on failure
        return "";
    }

    void Interrupt(BenchmarkState *state) override {
        // Handle timeout
    }

    string BenchmarkInfo() override {
        return "A custom benchmark for testing my feature";
    }

    string GetLogOutput(BenchmarkState *state) override {
        return "";
    }
};

// File-scope instance triggers self-registration
MyCustomBenchmark my_custom_benchmark_instance;

The benchmark is automatically discovered by the runner because the file-scope instance calls the Benchmark constructor with register_benchmark=true, which adds it to the global registry before main() executes.

Related

• Principle:Duckdb_Duckdb_Benchmark_Discovery