Heuristic:ClickHouse ClickHouse Debug Build Tips
| Knowledge Sources | |
|---|---|
| Domains | Debugging, Build_System |
| Last Updated | 2026-02-08 18:00 GMT |
Overview
Collection of debugging-related build tips for ClickHouse: using `DEBUG_O_LEVEL=0` to see variables in debugger, understanding `OMIT_HEAVY_DEBUG_SYMBOLS` behavior, configuring Ninja parallelism, and recognizing platform-specific profiler limitations.
Description
ClickHouse has several build-time configurations that affect debuggability. The default debug optimization level is `-Og` which still optimizes some variables away; setting `DEBUG_O_LEVEL=0` preserves all variables at the cost of slower execution. Heavy modules (Functions, Dictionaries, Arrow, AWS) are compiled with `-g1` instead of full debug info in release builds to reduce binary size and link time. The Query Profiler requires `-fasynchronous-unwind-tables` (enabled on Linux, unavailable on macOS). Build parallelism should never be manually specified with `ninja -j`; let Ninja auto-tune based on available resources.
Usage
Apply this heuristic when setting up a development build, debugging with gdb/lldb, or investigating slow profiler output. These tips address the most common frustrations new developers encounter.
The Insight (Rule of Thumb)
- Action 1 (Optimized Out Variables): If gdb/lldb shows `<optimized out>` for variables, rebuild with `-DDEBUG_O_LEVEL="0"`. Default `-Og` still optimizes.
- Value: Makes all local variables visible in debugger at the cost of 2-3x slower debug execution.
- Trade-off: `-O0` produces much slower code than `-Og`, but full variable visibility is worth it when actively debugging.
- Action 2 (Heavy Debug Symbols): In RelWithDebInfo builds, modules like Functions, Dictionaries, and some contribs use `-g1` (minimal debug info). For full debug info in these modules, set `-DOMIT_HEAVY_DEBUG_SYMBOLS=OFF`.
- Value: Reduces link time by 30-50% and binary size by several GB.
- Trade-off: Stack frames in heavy modules show as empty (no local variables) when this is ON.
- Action 3 (Ninja Parallelism): Never use `ninja -j N`. Let Ninja auto-tune based on `cmake/limit_jobs.cmake` calculations (2500MB per compile job, 5000MB per link job).
- Value: Prevents OOM kills during linking and optimizes for the actual available memory.
- Trade-off: None. Manual `-j` overrides are strictly worse.
- Action 4 (Query Profiler): On macOS, the Query Profiler (stacktrace sampling) does not work because PHDR cache is unavailable and stacktrace collection is not async-signal-safe. Use Linux for profiling.
- Value: Knowing this limitation saves hours of debugging "why profiler output is empty."
- Action 5 (Function Alignment): `-falign-functions=64` and `-mbranches-within-32B-boundaries` (AMD64) prevent random benchmark noise from code layout changes. Do not disable these flags when benchmarking.
- Value: Benchmark results are reproducible across code changes that don't affect the measured code path.
Reasoning
DEBUG_O_LEVEL from `CMakeLists.txt:318-320`:
# Before you start hating your debugger because it refuses to show variables
# ('<optimized out>'), try building with -DDEBUG_O_LEVEL="0"
set(DEBUG_O_LEVEL "g" CACHE STRING "The -Ox level used for debug builds")
OMIT_HEAVY_DEBUG_SYMBOLS from `CMakeLists.txt:179-184`:
# Provides faster linking and lower binary size.
# Tradeoff is the inability to debug some source files with e.g. gdb
# (empty stack frames and no local variables).
option(OMIT_HEAVY_DEBUG_SYMBOLS
"Do not generate debugger info for heavy modules" ${OMIT_HEAVY_DEBUG_SYMBOLS_DEFAULT})
Query Profiler limitation from `CMakeLists.txt:232-241`:
# Query Profiler doesn't work on MacOS for several reasons
# - PHDR cache is not available
# - We use native functionality to get stacktraces which is not async signal safe
if (NOT OS_DARWIN)
set (COMPILER_FLAGS "${COMPILER_FLAGS} -fasynchronous-unwind-tables")
endif()
Ninja auto-tuning from `.claude/CLAUDE.md`:
Do not use `-j` argument with ninja - let it decide automatically.
Benchmark alignment from `CMakeLists.txt:279-287`:
# falign-functions=64 prevents from random performance regressions with the
# code change. Thus, providing more stable benchmarks.
set(COMPILER_FLAGS "${COMPILER_FLAGS} -falign-functions=64")