Principle:Duckdb Duckdb Benchmark Discovery

Overview

Benchmark Discovery is the principle of automatic registration and enumeration of performance benchmarks within a testing framework. Rather than requiring a central manifest or manual registration of each benchmark, the system allows benchmark instances to self-register into a global registry at construction time. This enables the benchmark runner to discover all available benchmarks without explicit coordination, supporting extensibility and reducing the maintenance burden of adding new benchmarks.

Description

The core mechanism behind Benchmark Discovery is self-registration via static initialization. Each benchmark object registers itself with a global registry when it is constructed, typically as a static or file-scope variable. At runtime, the benchmark runner queries this registry to obtain the full list of available benchmarks.

Key aspects of this principle include:

• Self-registering constructors -- Each benchmark class constructor accepts a register_benchmark flag. When true, the constructor adds the benchmark instance to a global list during object initialization, before main() is invoked.
• Global registry pattern -- A centralized data structure (typically a list or map) holds pointers to all registered benchmark instances. This registry is the single source of truth for benchmark enumeration.
• Name and group metadata -- Each benchmark provides a name and group string at registration time. These metadata fields enable filtering, categorization, and selective execution of benchmarks.
• Decentralized definition -- Benchmarks are defined across many source files. The discovery mechanism eliminates the need for a central file that lists all benchmarks, allowing each benchmark to be self-contained.
• Enumeration before execution -- The runner can list all discovered benchmarks (e.g., via a --list flag) before deciding which ones to execute. This supports filtering by name pattern, group, or other criteria.

Usage

Apply Benchmark Discovery when:

• A benchmark framework needs to support an extensible set of benchmarks without requiring changes to a central registry file.
• New benchmarks should be automatically available to the runner simply by being compiled and linked into the binary.
• Benchmark enumeration is needed as a preliminary step before execution, for example to support --list or pattern-based filtering.
• The project contains benchmarks spread across many files and directories, making manual registration impractical.

Theoretical Basis

This principle draws on several well-established design patterns:

• Factory pattern -- The global registry acts as a factory that can produce or enumerate benchmark instances. Clients query the registry rather than constructing benchmarks directly.
• Self-registration via static initialization -- A common C++ idiom where file-scope objects register themselves with a global during construction. This relies on the guarantee that static storage duration objects are initialized before main() begins.
• Service locator pattern -- The registry serves as a service locator: benchmark consumers (the runner) locate benchmark providers (individual benchmark classes) through a well-known registry rather than through direct coupling.
• Convention over configuration -- By adopting the convention that benchmarks self-register, the framework eliminates the configuration step of maintaining a manual benchmark list.

Related

• Implementation:Duckdb_Duckdb_Benchmark_Class