Implementation:MaterializeInc Materialize CLI Optbench
| Knowledge Sources | |
|---|---|
| Domains | CLI, Benchmarking, Query Optimization |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
The CLI Optbench tool benchmarks SQL query optimizer performance by measuring EXPLAIN plan generation times across scenarios, supporting comparison between Materialize and PostgreSQL.
Description
This module provides the optbench Click-based CLI application for measuring query optimization times. It loads predefined benchmark scenarios, executes EXPLAIN statements against a target database (Materialize or PostgreSQL), collects optimization timing data using numpy/pandas, and supports comparing results between a baseline and a target run. Results are stored as CSV files with version-specific naming conventions. The tool integrates with the materialize.optbench package for scenario definitions, SQL execution, and utility functions.
Usage
Use this tool to measure and compare query optimizer performance across Materialize versions or between Materialize and PostgreSQL, identifying optimization regressions or improvements.
Code Reference
Source Location
- Repository: MaterializeInc_Materialize
- File: misc/python/materialize/cli/optbench.py
Signature
@click.group()
def app() -> None: ...
class Arg:
scenario = dict(type=click.Choice(scenarios()), callback=_scenario_callback)
base = dict(type=click.Path(...), callback=_path_callback)
Import
from materialize.cli.optbench import app
I/O Contract
| Input | Type | Description |
|---|---|---|
| scenario | Scenario |
Benchmark scenario name from predefined set |
| base | Path |
Path to baseline CSV results for comparison |
| database port | int |
Target database connection port |
| Output | Type | Description |
|---|---|---|
| CSV results | file | Optimization timing measurements per query per scenario |
| comparison | stdout | Statistical comparison of base vs target optimization times |
Usage Examples
# Run an optimization benchmark
optbench run --scenario tpch --port 6875
# Compare results against a baseline
optbench compare --base results-baseline.csv --target results-new.csv