Principle:ClickHouse ClickHouse Third Party Dependency Management

Knowledge Sources	ClickHouse CMake add_subdirectory Documentation
Domains	Build_System, C++, Dependency_Management
Last Updated	2026-02-08 00:00 GMT

Overview

Vendored dependency compilation with a CMake wrapper pattern is a technique where a project compiles third-party libraries from source using its own custom CMake files rather than each library's native build system.

Description

ClickHouse vendors approximately 90 third-party libraries in its contrib/ directory as Git submodules. Rather than using each library's own build system (which may use CMake, autotools, Meson, Bazel, or other tools), ClickHouse provides its own CMake wrapper files that compile only the needed source files with the project's own compiler flags and settings.

This approach is implemented through a two-directory convention:

• Source directory: contrib/{library-name}/ contains the unmodified upstream source code (as a Git submodule).
• CMake wrapper directory: contrib/{library-name}-cmake/ contains a custom CMakeLists.txt that knows which source files to compile and which compile options to set.

For example, OpenSSL's source lives in contrib/openssl/ while the CMake wrapper that builds it lives in contrib/openssl-cmake/. This separation keeps the upstream source pristine while giving full control over the build process.

The benefits of this approach are significant:

• Uniform compiler flags: All dependencies are compiled with the same compiler, standard library, optimization level, and sanitizer settings as the main project. This eliminates subtle ABI mismatches.
• Selective compilation: Only the source files actually needed are compiled. Many large libraries (like Boost, LLVM, or Arrow) contain far more code than ClickHouse uses.
• No external tool dependencies: The build requires only CMake, Ninja, and a C/C++ compiler. No autotools, Meson, Bazel, or other build systems are needed.
• Sanitizer compatibility: When building with ASan, MSan, TSan, or UBSan, all dependencies are instrumented, providing full coverage for bug detection.

The contrib/CMakeLists.txt file suppresses warnings in third-party code (-w flag) and enables function/data sections (-ffunction-sections -fdata-sections) to allow the linker to garbage-collect unused code.

Usage

Use the CMake wrapper pattern when:

• The project needs all dependencies compiled with identical compiler flags and sanitizer instrumentation.
• Upstream build systems are complex, fragile, or introduce unwanted dependencies.
• Only a subset of each library's functionality is needed.
• The project must support cross-compilation where native build systems may not work correctly.
• Reproducibility requires deterministic builds without relying on system-installed library versions.

Theoretical Basis

The CMake wrapper pattern relies on CMake's add_subdirectory command, which processes a child CMakeLists.txt in a specified directory. The add_contrib function in ClickHouse adds a validation layer on top of this:

function add_contrib(cmake_folder, base_folder1, ..., base_folderN):
    if no base_folders specified:
        base_folders = [cmake_folder]

    for each base_folder in base_folders:
        if base_folder does not exist:
            FATAL_ERROR("Typo in folder name?")
        if base_folder is empty:
            WARNING("Submodule missing, run: git submodule update --init")
            return  (skip this dependency gracefully)

    add_subdirectory(cmake_folder)  # Process the CMake wrapper

This design handles two important cases:

1. Missing submodule (typo): If the directory does not exist at all, it is a fatal error, likely indicating a typo in the CMakeLists.txt.
2. Empty submodule (not initialized): If the directory exists but is empty (common in CI fast-test builds that only initialize a minimal subset of submodules), the dependency is skipped with a warning rather than failing the build.

The function accepts a cmake_folder (the wrapper directory) and one or more base_folder arguments (the actual submodule source directories). When the source and wrapper directories differ (the common case), both are specified:

add_contrib (openssl-cmake openssl)        # wrapper in openssl-cmake/, source in openssl/
add_contrib (arrow-cmake arrow)            # wrapper in arrow-cmake/, source in arrow/
add_contrib (consistent-hashing)           # wrapper and source in same directory

Some complex dependencies declare multiple base folders to verify that all required submodules are present. For example, the AWS SDK wrapper verifies over a dozen submodule directories.

Related Pages

Implemented By

• Implementation:ClickHouse_ClickHouse_Add_Contrib_Macro