Implementation:ClickHouse ClickHouse Clickhouse Test Runner

Knowledge Sources	ClickHouse
Domains	Testing
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for executing ClickHouse stateless functional tests, provided by the clickhouse-test Python script.

Description

clickhouse-test is a 5,270-line Python 3 script that orchestrates the execution of stateless tests against a running ClickHouse server. It discovers test files in tests/queries/0_stateless/, executes each one according to its format (.sql, .sql.j2, .sh, .py, .expect), captures the output, and compares it against the corresponding .reference file. Tests are classified as OK, FAIL, SKIPPED, UNKNOWN, or NOT_FAILED based on the comparison result and other checks.

Key architectural components of the script:

run_single_test (line 2582): Executes one test case. Handles process spawning, timeout enforcement, output capture, sanitizer log detection, and reference comparison.
run_tests_array (line 3389): Manages the execution of a batch of tests, coordinating parallelism, retry logic, and shared state across worker processes.
main (line 4139): Entry point that parses arguments, verifies the server is alive (via SELECT 1), loads blacklists, discovers tests, and dispatches execution.

The runner supports automatic retry: when a test fails, it can be retried up to MAX_RETRIES = 3 times (defined at line 82). The default per-test timeout is TEST_MAX_RUN_TIME_IN_SECONDS = 180 seconds (line 88).

The runner communicates with the ClickHouse server using the environment variables CLICKHOUSE_PORT_TCP and CLICKHOUSE_PORT_HTTP to locate the server's native and HTTP endpoints respectively.

Usage

Use this tool after starting a ClickHouse server from a development build. It is the primary mechanism for validating SQL correctness during local development and in CI pipelines.

Code Reference

Source Location

Repository: ClickHouse
File: tests/clickhouse-test (lines 1-5270)
Key functions:
- run_single_test -- line 2582
- run_tests_array -- line 3389
- main -- line 4139
- check_server_started -- line 3589

Signature

CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test [test_pattern] [options]

I/O Contract

Inputs

Name	Type	Required	Description
`test`	Positional argument (regex string)	No	Regular expression pattern for selecting tests by name. If omitted, all tests are run
`--no-random-settings`	Flag	No	Disables randomization of ClickHouse settings between test runs. Useful for reproducing failures deterministically
`--no-random-merge-tree-settings`	Flag	No	Disables randomization of MergeTree-specific settings
`--record`	Flag	No	Automatically updates `.reference` files with the actual output of each test. Used when intentionally changing expected behavior
`-j` / `--jobs`	Integer	No	Number of parallel worker processes. Defaults to automatic detection
`--timeout`	Integer (seconds)	No	Per-test timeout. Default is 180 seconds (`TEST_MAX_RUN_TIME_IN_SECONDS`)
`--hung-check`	Flag	No	Enables detection of hung queries and prints server stack traces on timeout
`--test-runs`	Integer	No	Number of times to repeat each test
`--order`	String	No	Controls test execution order (e.g., `random`, `asc`, `desc`)
Running ClickHouse server	Network service	Yes	A server listening on the ports specified by `CLICKHOUSE_PORT_TCP` and `CLICKHOUSE_PORT_HTTP`
Test files	Files in `tests/queries/0_stateless/`	Yes	The test scripts and their corresponding `.reference` files

Outputs

Name	Type	Description
Per-test result	Console output	Each test is reported as OK, FAIL, SKIPPED, UNKNOWN, or NOT_FAILED with timing information
Exit code	Integer	0 if all tests passed, non-zero otherwise
Updated reference files	Files (when `--record` is used)	`.reference` files overwritten with actual test output
Retry attempts	Console output	Failed tests are automatically retried up to `MAX_RETRIES = 3` times before being reported as final failures

Usage Examples

Run a single test by name:

CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test 01234_my_test

Run all tests with deterministic settings:

CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test \
    --no-random-settings --no-random-merge-tree-settings

Run tests matching a regex pattern in parallel:

CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test \
    "02.*aggregat" -j 4

Record new reference files after an intentional behavior change:

CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test \
    01234_my_test --record

Run with hung check enabled (useful for debugging stuck queries):

CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test \
    01234_my_test --hung-check

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment