Principle:ClickHouse ClickHouse Debug Package Configuration
| Knowledge Sources | |
|---|---|
| Domains | Packaging, Distribution |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Debug package configuration is the practice of distributing debug symbols as separate, optional packages that can be installed alongside release binaries to enable post-mortem analysis and live debugging on production systems.
Description
After debug symbol splitting produces a stripped binary and a separate .debug file, the debug file must be distributed to systems where debugging may be needed. Rather than including these large files in the main binary package, they are placed in dedicated debug packages.
ClickHouse produces two debug packages:
clickhouse-common-static-dbg: Contains debug symbols for the mainclickhousebinary. This is the primary debug package, as the single ClickHouse binary contains all functionality (server, client, local, etc.).clickhouse-keeper-dbg: Contains debug symbols for the standaloneclickhouse-keeperbinary, used when Keeper is built and distributed as a separate executable.
Both packages install their .debug files into the standard GNU debug directory at /usr/lib/debug/usr/bin/, where debuggers such as GDB automatically discover them.
The debug packages are marked as optional: the main package (clickhouse-common-static) suggests the debug package, meaning it will not be installed by default but package managers will display it as a recommendation.
Usage
Debug package configuration should be applied when:
- Building ClickHouse release packages to provide operators with the option of installing debug symbols.
- Investigating production exceptions by installing the debug package matching the deployed version and then analyzing core dumps with GDB.
- Profiling or tracing ClickHouse in production, where debug symbols provide function names and line numbers for tools like
perf. - Building packages without full debug info: The build system supports creating empty placeholder
.debugfiles for nfpm packaging when the full debug information is not available or not needed.
Theoretical Basis
Standard Debug File Locations
The GNU debugging conventions define a hierarchy for locating separate debug files. The primary mechanism used by ClickHouse is the global debug directory, where debug files are placed at a path that mirrors the binary's install location under /usr/lib/debug/:
/usr/bin/clickhouse (stripped binary)
/usr/lib/debug/usr/bin/clickhouse.debug (debug symbols)
When GDB loads /usr/bin/clickhouse, it checks for a .gnu_debuglink section pointing to clickhouse.debug and searches the global debug directory. If the file exists and its CRC32 matches, the symbols are loaded transparently.
Build ID as an Alternative Lookup
Modern ELF binaries also contain a build ID (a unique hash in the .note.gnu.build-id section). Debuggers can use this as an alternative lookup path:
/usr/lib/debug/.build-id/ab/cdef1234567890.debug
ClickHouse primarily relies on the gnu_debuglink mechanism but the build ID provides an additional fallback for tools that support it.
Empty Placeholder Debug Files
The ClickHouse build system includes a mechanism (clickhouse_make_empty_debug_info_for_nfpm) to create empty .debug files. This is necessary because nfpm requires all files listed in the package YAML to exist at build time, even when:
- The build configuration does not produce debug symbols (e.g., release builds without
-g). - The debug package is being built as a structural placeholder that will be populated later.
- CI pipelines build packages in stages, where the binary and debug info may be produced separately.
The empty file ensures the package can always be built, even if the resulting debug package provides no actual debugging capability.
Separation of Concerns
Distributing debug symbols as separate packages follows the principle of separation of concerns:
- Base package (
clickhouse-common-static): ~500 MB stripped binary. Installed on all systems. - Debug package (
clickhouse-common-static-dbg): Multiple GB of debug data. Installed only when needed.
This separation keeps production systems lean while preserving full debuggability as an opt-in capability. The strict version correspondence between base and debug packages ensures that symbols always match the binary they describe.