Workflow:ClickHouse ClickHouse Running Stateless Tests
| Knowledge Sources | |
|---|---|
| Domains | Testing, Quality_Assurance, Database_Engineering |
| Last Updated | 2026-02-08 17:00 GMT |
Overview
End-to-end process for building ClickHouse, starting a local server, and executing stateless functional tests to validate query behavior and correctness.
Description
This workflow covers the standard developer testing loop for ClickHouse. Stateless tests are the primary validation mechanism, located in `tests/queries/0_stateless/`. Each test consists of a query file (`.sql`, `.sh`, `.py`, or `.expect`) and a reference file (`.reference`) containing expected output. The test runner creates a temporary database for isolation, executes the test, and compares stdout against the reference file. Tests support tags for controlling execution (e.g., `no-parallel`, `no-fasttest`, `disabled`), randomized settings for robustness, and Jinja2 templating for parameterized tests.
Usage
Execute this workflow whenever you modify ClickHouse source code and need to verify that existing functionality is not broken, or when developing new features that require test coverage. This is the mandatory validation step before submitting any pull request.
Execution Steps
Step 1: Build ClickHouse
Compile ClickHouse using Ninja to produce the `clickhouse` binary. The build must succeed before any tests can run. For development, a RelWithDebInfo or Debug build is typical. Sanitizer builds (ASan, TSan) can be used for detecting memory or concurrency issues during tests.
Key considerations:
- Build command: `cd build && ninja clickhouse`
- Do not pass `-j` to Ninja; let it auto-detect parallelism
- Different build directories can be used: `build/`, `build_debug/`, `build_asan/`
Step 2: Start the ClickHouse Server
Launch a local ClickHouse server instance using the development configuration. The server must be fully initialized and accepting connections before tests can begin. Use the default configuration file from the source tree.
Key considerations:
- Start command: `./build/programs/clickhouse server --config-file ./programs/server/config.xml`
- Default ports: TCP 9000, HTTP 8123
- Wait for readiness by polling: `./build/programs/clickhouse client -q "SELECT 1"`
- The server runs in the foreground; use a separate terminal or background process
Step 3: Execute Stateless Tests
Run the test suite using the `clickhouse-test` runner with appropriate port environment variables. Tests can be run individually by name or as the full suite. The runner handles database creation, output capture, reference comparison, and result reporting.
Key considerations:
- Run command: `CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test <test_name>`
- Useful flags: `--no-random-settings`, `--no-random-merge-tree-settings`, `--record`
- The `--record` flag updates reference files automatically when output differs
- Test files use extensions: `.sql`, `.sh`, `.py`, `.expect`, `.sql.j2`
- Reference files use `.reference` or `.gen.reference` (for Jinja2 tests)
Step 4: Analyze Test Results
Review test output to identify failures. Failed tests show a diff between actual and expected output. The test runner normalizes database names (replacing random temporary names with `default`) before comparison. Check for regressions in existing tests and validate new test coverage.
Key considerations:
- Random database names are normalized to `default` in output comparison
- Reference files should use `default` for database names, not `${CLICKHOUSE_DATABASE}`
- Tests tagged with `disabled` are skipped by default
- Blacklist files exist for tests incompatible with specific configurations (async inserts, parallel replicas)
- Sanitizer ignorelists (`tests/tsan_ignorelist.txt`, `tests/ubsan_ignorelist.txt`) suppress known false positives
Step 5: Stop the Server
Cleanly shut down the local ClickHouse server after testing is complete. Find the server process and send a termination signal.
Key considerations:
- Find PIDs: `pgrep -f "clickhouse server"`
- Stop: `kill <pid>`
- Ensure the server has fully stopped before restarting with different configuration