Implementation:ClickHouse ClickHouse Find Symbols

Knowledge Sources	ClickHouse
Domains	String_Processing, SIMD
Last Updated	2026-02-08 00:00 GMT

Overview

SIMD-accelerated character search functions that locate specific characters or character sets in strings much faster than standard library alternatives.

Description

This code provides a family of functions for searching characters in strings, similar to `strpbrk`, `strcspn`, and `strchr` but optimized with SIMD instructions. The implementation uses SSE2 for small character sets and SSE4.2 for larger sets, providing 2-10x performance improvements over naive loops. Functions can search for the first occurrence, last occurrence, or count occurrences of characters from a set. The API supports both compile-time character sets (via template parameters) and runtime sets (via `SearchSymbols` struct). The functions work with non-null-terminated memory ranges and return pointers to found characters or null/end markers.

Usage

Use this when parsing delimited data formats (CSV, TSV, logs), searching for special characters in strings (quotes, escapes, delimiters), or implementing custom string tokenization. The SIMD optimizations provide significant speedups when processing large volumes of text data, such as parsing database dumps or log files.

Code Reference

Source Location

• Repository: ClickHouse
• File: base/base/find_symbols.h
• Lines: 1-530

Signature

// Search for first occurrence of any character in set
template <char... symbols>
inline const char * find_first_symbols(const char * begin, const char * end);

// Runtime character set version
inline const char * find_first_symbols(std::string_view haystack, const SearchSymbols & symbols);

// Search for first character NOT in set
template <char... symbols>
inline const char * find_first_not_symbols(const char * begin, const char * end);

// Return nullptr if not found (instead of end)
template <char... symbols>
inline const char * find_first_symbols_or_null(const char * begin, const char * end);

// Search backwards for last occurrence
template <char... symbols>
inline const char * find_last_symbols_or_null(const char * begin, const char * end);

// Count occurrences of characters in set
template <char... symbols>
inline size_t count_symbols(const char * begin, const char * end);

// Split string on delimiters
template <char... symbols, typename To>
inline To & splitInto(To & to, std::string_view what, bool token_compress = false);

// SearchSymbols for runtime character sets
struct SearchSymbols {
    static constexpr auto BUFFER_SIZE = 16;
    explicit SearchSymbols(std::string in);
    std::string str;
#if defined(__SSE4_2__)
    __m128i simd_vector;
#endif
};

Import

#include <base/find_symbols.h>

I/O Contract

Inputs

Name	Type	Required	Description
begin	const char *	Yes	Start of search range
end	const char *	Yes	End of search range (non-inclusive)
haystack	std::string_view	Yes	String to search (alternative interface)
symbols	SearchSymbols	Yes	Runtime character set
token_compress	bool	No	Skip empty tokens in split (default: false)

Outputs

Name	Type	Description
position	const char *	Pointer to found character, or end/nullptr
count	size_t	Number of matching characters
tokens	Container	Split string parts (for `splitInto`)

Usage Examples

#include <base/find_symbols.h>
#include <vector>

// Find delimiter in CSV parsing
const char * line = "apple,banana,cherry";
const char * comma = find_first_symbols<','>(line, line + strlen(line));
std::string first_field(line, comma);  // "apple"

// Find quote or escape character
const char * text = R"(He said \"Hello\")";
const char * special = find_first_symbols<'"', '\\'>(text, text + strlen(text));

// Find end of whitespace
const char * str = "   hello";
const char * non_ws = find_first_not_symbols<' ', '\t', '\n'>(str, str + strlen(str));
std::string trimmed(non_ws, str + strlen(str));  // "hello"

// Count newlines in text
const char * data = "line1\nline2\nline3\n";
size_t line_count = count_symbols<'\n'>(data, data + strlen(data)) + 1;

// Split string on multiple delimiters
std::vector<std::string_view> parts;
splitInto<',', ';'>(parts, "a,b;c,d", false);
// parts = {"a", "b", "c", "d"}

// Runtime character set
SearchSymbols delimiters(",;|\t");
const char * csv = "a,b;c|d\te";
const char * delim = find_first_symbols(std::string_view(csv), delimiters);

// Find last occurrence
const char * path = "/usr/local/bin/program";
const char * last_slash = find_last_symbols_or_null<'/'>(path, path + strlen(path));
std::string filename(last_slash + 1, path + strlen(path));  // "program"

// Tokenize with compression (skip empty tokens)
std::vector<std::string_view> tokens;
splitInto<' '>(tokens, "word1  word2   word3", true);  // Skips empty tokens
// tokens = {"word1", "word2", "word3"}

Related Pages

• Principle:ClickHouse_ClickHouse_SIMD_Character_Search