Implementation:ClickHouse ClickHouse Test Blacklist Reference Comparison

Knowledge Sources	ClickHouse
Domains	Testing
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete pattern for categorizing test results via blacklist files and sanitizer ignorelists, internal to the clickhouse-test runner and the CMake build system.

Description

The ClickHouse test infrastructure uses two complementary mechanisms to manage known issues:

1. Test-level blacklists are plain text files listing test names (one per line) that are expected to fail under certain run modes. The clickhouse-test runner loads these files during initialization (lines 4298-4314 of tests/clickhouse-test) and uses them to classify results:

tests/analyzer_tech_debt.txt -- Always loaded. Contains tests that are known to fail due to analyzer-related technical debt.
tests/parallel_replicas_blacklist.txt (409 lines) -- Loaded when the --no-parallel-replicas flag is active. Lists tests incompatible with parallel replicas mode.
tests/async_insert_blacklist.txt (52 lines) -- Loaded when the --no-async-insert flag is active. Lists tests incompatible with async insert mode.

When a test succeeds but its name appears in a loaded blacklist, the runner assigns the NOT_FAILED status (defined at lines 1085-1088):

NOT_FAILED = (
    "The test succeeded, but it is listed in parallel_replicas_blacklist.txt or async_insert_blacklist.txt. "
    "Please remove it from the list"
)

The five possible test statuses are defined in the TestStatus enum (lines 1026-1031):

class TestStatus(enum.Enum):
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"
    OK = "OK"
    SKIPPED = "SKIPPED"
    NOT_FAILED = "NOT_FAILED"

2. Sanitizer ignorelists are compiler-level files consumed at build time via cmake/sanitize.cmake:

tests/tsan_ignorelist.txt (13 lines) -- Suppresses ThreadSanitizer instrumentation for specific functions. Referenced at cmake/sanitize.cmake:L30 via -fsanitize-ignorelist.
tests/ubsan_ignorelist.txt (22 lines) -- Suppresses UndefinedBehaviorSanitizer instrumentation for specific source files. Referenced at cmake/sanitize.cmake:L46 via -fsanitize-ignorelist.

These files use Clang's Sanitizer Special Case List format and are distinct from runtime suppression files.

Usage

Use this pattern when you need to understand why a test is categorized with a particular status, when adding or removing entries from blacklist files, or when investigating sanitizer reports that appear to be false positives.

Code Reference

Source Location

Repository: ClickHouse
Blacklist loading: tests/clickhouse-test lines 4298-4314
NOT_FAILED status: tests/clickhouse-test lines 1085-1088
TestStatus enum: tests/clickhouse-test lines 1026-1031
Sanitizer CMake config: cmake/sanitize.cmake lines 30 (TSan) and 46 (UBSan)

Blacklist Loading Logic

# tests/clickhouse-test lines 4298-4314 (simplified)

blacklist_check = []

# Always loaded
blacklist_check.extend(try_get_skip_list(base_dir, "../analyzer_tech_debt.txt"))

# Conditional on --no-parallel-replicas
if args.no_parallel_replicas is True:
    blacklist = try_get_skip_list(base_dir, "../parallel_replicas_blacklist.txt")
    blacklist_check.extend(blacklist)

# Conditional on --no-async-insert
if args.no_async_insert is True:
    skip_list_async_inserts = try_get_skip_list(base_dir, "../async_insert_blacklist.txt")
    blacklist_check.extend(skip_list_async_inserts)

Sanitizer Ignorelist CMake Integration

# cmake/sanitize.cmake

# TSan (line 30)
set (TSAN_FLAGS "${TSAN_FLAGS} -fsanitize-ignorelist=${PROJECT_SOURCE_DIR}/tests/tsan_ignorelist.txt")

# UBSan (line 46)
set (UBSAN_FLAGS "${UBSAN_FLAGS} -fsanitize-ignorelist=${PROJECT_SOURCE_DIR}/tests/ubsan_ignorelist.txt")

I/O Contract

Inputs

Name	Type	Required	Description
Test execution results	Per-test status from runner	Yes	The raw pass/fail outcome of each test after reference comparison
`tests/analyzer_tech_debt.txt`	Text file (test names, one per line)	Yes (always loaded)	Tests with known analyzer-related failures
`tests/parallel_replicas_blacklist.txt`	Text file (409 lines)	Conditional (`--no-parallel-replicas`)	Tests incompatible with parallel replicas mode
`tests/async_insert_blacklist.txt`	Text file (52 lines)	Conditional (`--no-async-insert`)	Tests incompatible with async insert mode
`tests/tsan_ignorelist.txt`	Clang ignorelist file (13 lines)	Build time only	Functions excluded from ThreadSanitizer instrumentation
`tests/ubsan_ignorelist.txt`	Clang ignorelist file (22 lines)	Build time only	Source files excluded from UBSan instrumentation

Outputs

Name	Type	Description
OK	Test status	Test passed and is not blacklisted
FAIL	Test status	Test failed (output mismatch, timeout, server died, non-zero exit code, or stderr output)
SKIPPED	Test status	Test was not executed due to unmet preconditions (build type, missing feature, tag exclusion)
UNKNOWN	Test status	Result could not be determined (no reference file, internal runner error)
NOT_FAILED	Test status	Test passed but is listed in a blacklist -- the entry should be removed as technical debt cleanup

Usage Examples

Identify NOT_FAILED tests (stale blacklist entries) in CI output:

# Look for NOT_FAILED in the test runner output
CLICKHOUSE_PORT_TCP=9000 CLICKHOUSE_PORT_HTTP=8123 ./tests/clickhouse-test \
    --no-parallel-replicas 2>&1 | grep "NOT_FAILED"

Inspect the parallel replicas blacklist:

# View which tests are blacklisted for parallel replicas mode
head -20 tests/parallel_replicas_blacklist.txt

Check the TSan ignorelist contents:

cat tests/tsan_ignorelist.txt

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment