Principle:ClickHouse ClickHouse Test Result Analysis
| Knowledge Sources | |
|---|---|
| Domains | Testing, Quality_Assurance |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Categorizing test outcomes using blacklists and sanitizer ignorelists to distinguish genuine failures from known issues and false positives.
Description
After the test runner executes each test, the result must be categorized. ClickHouse uses a multi-tier classification system that goes beyond simple pass/fail. The five test statuses are:
- OK -- The test passed: its output matches the reference file.
- FAIL -- The test failed: output differs from the reference, a timeout occurred, the server died, or another error was detected.
- SKIPPED -- The test was not executed because a precondition was not met (e.g., a required feature is disabled, the build type does not support it, or a tag in the test file excludes it from the current run).
- UNKNOWN -- The test result could not be determined, for example because there is no reference file or an internal error occurred in the runner.
- NOT_FAILED -- The test succeeded, but it is listed in a blacklist file. This status signals technical debt: the blacklist entry should be removed because the test is no longer failing.
The blacklist mechanism allows ClickHouse to track known-failing tests without blocking the entire test suite. When a test mode (such as parallel replicas or async inserts) is enabled, certain tests are expected to fail. Rather than marking them as permanent SKIPs, they are blacklisted so that:
- Genuine new failures in these tests are still detected if the failure mode changes.
- When a fix lands, the
NOT_FAILEDstatus signals that the blacklist entry is stale and should be cleaned up.
Separately, sanitizer ignorelists suppress false-positive reports from Clang sanitizers (TSan, UBSan) at build time. These are not test-level blacklists but rather compiler-level directives that prevent instrumentation of specific functions or source files known to trigger benign warnings.
Usage
Use test result analysis whenever you need to:
- Understand why a test is reported with a particular status in CI.
- Determine whether a failure is a known issue (blacklisted) or a new regression.
- Clean up technical debt by removing tests from blacklists when they begin passing.
- Investigate sanitizer reports that may be suppressed by ignorelists.
Theoretical Basis
Large software projects accumulate tests that fail intermittently or fail only under specific configurations. A naive pass/fail model forces developers to either fix all failures before merging (impractical at scale) or ignore all failures (dangerous). The blacklist approach provides a middle ground:
- Known failures are documented: each blacklist file serves as a ledger of technical debt.
- New failures are surfaced: a test not on the blacklist that fails is a genuine regression.
- Stale entries are detected: the
NOT_FAILEDstatus actively notifies maintainers when a previously broken test has been fixed.
Sanitizer ignorelists operate at a different layer. Clang sanitizers instrument compiled code to detect undefined behavior, data races, and memory errors at runtime. Third-party libraries and certain known-benign patterns can trigger false positives. Rather than disabling the sanitizer entirely, ignorelists selectively suppress instrumentation for specific functions or files, preserving coverage for the rest of the codebase. The ignorelist files are consumed at build time via the -fsanitize-ignorelist compiler flag, configured in cmake/sanitize.cmake.