Implementation:Duckdb Duckdb Zstd Common
| Knowledge Sources | |
|---|---|
| Domains | Compression, Third_Party |
| Last Updated | 2026-02-07 12:00 GMT |
Overview
Zstd Common contains the shared utility libraries, entropy coding primitives, hashing, threading infrastructure, and public header definitions used by both the Zstd compressor and decompressor within DuckDB.
Description
The common layer of the Zstd library in DuckDB (namespace duckdb_zstd) provides foundational components shared across all Zstd subsystems:
- zstd.h (3093 lines) -- the master public API header defining the Simple API, Explicit Context API, Streaming API, constants (magic numbers, block sizes, version information), and error handling. Zstd version 1.5.6 with compression levels from
ZSTD_minCLevel()(negative) throughZSTD_maxCLevel()(currently 22), default level 3. - fse.h (637 lines) -- Finite State Entropy (FSE) codec public prototypes (version 0.9.0). FSE is used for entropy coding of match lengths, literal lengths, and offsets. Provides
FSE_compressBound(), error management, and detailed compression/decompression pipelines in 5-step and 3-step processes respectively. - compiler.h (369 lines) -- compiler abstraction macros for portability across GCC, Clang, MSVC, and others; includes force-inline, target attribute, and sanitizer annotations.
- entropy_common.cpp (343 lines) -- shared entropy coding functions including
FSE_readNCount()for reading normalized frequency counts and version/error management for both FSE and Huffman coders. - fse_decompress.cpp (316 lines) -- FSE table decompression used during both zstd decompression and dictionary loading.
- pool.cpp (376 lines) -- thread pool implementation with job queue, mutex-based synchronization, and configurable thread count. Used by the multi-threaded compressor (
zstdmt_compress) and COVER dictionary trainer. - xxhash.cpp (863 lines) -- xxHash fast hashing (XXH32/XXH64) used for content checksums, dictionary ID hashing, and internal hash tables. Supports configurable memory access strategies (memcpy, packed, direct) for different architectures.
Usage
These common modules are compiled into every DuckDB build that includes Zstd support. They are not called directly by DuckDB application code but are linked by the Zstd compressor, decompressor, and dictionary modules. The zstd.h header is the primary include file for all Zstd functionality.
Code Reference
Source Location
- Repository: Duckdb_Duckdb
- Files:
- third_party/zstd/include/zstd.h (3093 lines) -- master Zstd public API header
- third_party/zstd/include/zdict.h (473 lines) -- dictionary builder API header
- third_party/zstd/include/zstd/common/fse.h (637 lines) -- FSE codec prototypes
- third_party/zstd/include/zstd/common/compiler.h (369 lines) -- compiler abstraction
- third_party/zstd/common/entropy_common.cpp (343 lines) -- shared entropy functions
- third_party/zstd/common/fse_decompress.cpp (316 lines) -- FSE table decompression
- third_party/zstd/common/pool.cpp (376 lines) -- thread pool
- third_party/zstd/common/xxhash.cpp (863 lines) -- xxHash implementation
Signature
namespace duckdb_zstd {
// --- zstd.h: Version and Constants ---
#define ZSTD_VERSION_MAJOR 1
#define ZSTD_VERSION_MINOR 5
#define ZSTD_VERSION_RELEASE 6
#define ZSTD_CLEVEL_DEFAULT 3
#define ZSTD_MAGICNUMBER 0xFD2FB528
#define ZSTD_MAGIC_DICTIONARY 0xEC30A437
ZSTDLIB_API unsigned ZSTD_versionNumber(void);
ZSTDLIB_API const char* ZSTD_versionString(void);
ZSTDLIB_API unsigned ZSTD_isError(size_t code);
ZSTDLIB_API const char* ZSTD_getErrorName(size_t code);
// --- fse.h: Finite State Entropy ---
#define FSE_VERSION_MAJOR 0
#define FSE_VERSION_MINOR 9
#define FSE_VERSION_RELEASE 0
FSE_PUBLIC_API size_t FSE_compressBound(size_t size);
FSE_PUBLIC_API unsigned FSE_isError(size_t code);
FSE_PUBLIC_API const char* FSE_getErrorName(size_t code);
FSE_PUBLIC_API unsigned FSE_versionNumber(void);
// --- entropy_common.cpp ---
unsigned FSE_isError(size_t code);
unsigned HUF_isError(size_t code);
const char* FSE_getErrorName(size_t code);
const char* HUF_getErrorName(size_t code);
size_t FSE_readNCount(short* normalizedCounter, unsigned* maxSVPtr,
unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize);
} // namespace duckdb_zstd
Import
#include "zstd.h" // master API
#include "zstd/common/fse.h" // FSE codec
#include "zstd/common/huf.h" // Huffman codec
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| code | size_t |
Yes | Return value from a Zstd/FSE/HUF function to test for errors |
| headerBuffer | const void* |
Yes (for FSE_readNCount) | Buffer containing serialized normalized frequency counts |
| hbSize | size_t |
Yes (for FSE_readNCount) | Size of the header buffer; must be >= 8 bytes for the optimized path |
| size | size_t |
Yes (for FSE_compressBound) | Uncompressed input size for computing maximum FSE-compressed output size |
Outputs
| Name | Type | Description |
|---|---|---|
| (return) from ZSTD_isError | unsigned |
1 if the code represents an error, 0 otherwise |
| (return) from ZSTD_getErrorName | const char* |
Human-readable error description string |
| (return) from FSE_compressBound | size_t |
Maximum compressed size for FSE encoding |
| normalizedCounter (out param) | short* |
Decoded frequency table from FSE_readNCount()
|
| maxSVPtr (out param) | unsigned* |
Maximum symbol value found in the table |
| tableLogPtr (out param) | unsigned* |
Log2 of the table size |
Usage Examples
#include "zstd.h"
using namespace duckdb_zstd;
// --- Version Check ---
unsigned version = ZSTD_versionNumber(); // e.g., 10506
const char* vstr = ZSTD_versionString(); // e.g., "1.5.6"
// --- Error Handling Pattern ---
size_t result = ZSTD_compress(dst, dstCap, src, srcSize, 3);
if (ZSTD_isError(result)) {
const char* errName = ZSTD_getErrorName(result);
// Handle error: errName is a readable string like "Src size is incorrect"
}
// --- Compression Level Bounds ---
int minLevel = ZSTD_minCLevel(); // negative value for fastest
int maxLevel = ZSTD_maxCLevel(); // 22 (ultra)
int defLevel = ZSTD_defaultCLevel(); // 3
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment