Implementation:Duckdb Duckdb Zstd Dictionary
| Knowledge Sources | |
|---|---|
| Domains | Compression, Third_Party |
| Last Updated | 2026-02-07 12:00 GMT |
Overview
Zstd Dictionary provides dictionary training functionality for the Zstandard compression library, enabling significantly improved compression ratios on small data by learning common patterns from representative samples.
Description
The Zstd dictionary builder in DuckDB, operating within the duckdb_zstd namespace, constructs optimized compression dictionaries from sample data. Dictionaries are most effective for compressing many small, structurally similar files (e.g., JSON records, database rows) where individual files are typically under 100 KB. A typical dictionary size is approximately 100 KB.
The dictionary training subsystem consists of four modules:
- zdict.cpp (1139 lines) -- main dictionary builder using a heuristic based on suffix-sorted frequency analysis; calls into FSE and Huffman encoders for entropy table construction; uses XXH64 hashing for content identification
- cover.cpp (1271 lines) -- COVER algorithm implementation based on the Liao-Petri-Moffat-Wirth paper (WWW 2016); builds dictionaries by selecting optimal segments from suffix-array-indexed sample data; supports multi-threaded optimization via thread pools
- fastcover.cpp (775 lines) -- fast approximation of the COVER algorithm with configurable acceleration factor; trades dictionary quality for faster training
- divsufsort.cpp (1916 lines) -- suffix array construction algorithm (SA-IS variant) used by the dictionary builder for efficient substring identification
A zstd dictionary consists of a header (magic number 0xEC30A437, dictionary ID, and pre-computed entropy tables) and content (raw byte sequences representing common patterns). The entropy tables in the header allow zstd to save on per-frame header costs, which is particularly beneficial for small data. Samples of approximately 100x the target dictionary size are recommended for training, with a minimum of 8 bytes per sample.
Usage
DuckDB includes the dictionary training modules to support advanced compression scenarios where repeated compression of small, similar data blocks benefits from shared dictionaries. The dictionaries are loaded into compression/decompression contexts via ZSTD_CCtx_loadDictionary() and ZSTD_DCtx_loadDictionary() respectively.
Code Reference
Source Location
- Repository: Duckdb_Duckdb
- Files:
- third_party/zstd/dict/zdict.cpp (1139 lines) -- main dictionary builder
- third_party/zstd/dict/cover.cpp (1271 lines) -- COVER algorithm
- third_party/zstd/dict/fastcover.cpp (775 lines) -- fast COVER algorithm
- third_party/zstd/dict/divsufsort.cpp (1916 lines) -- suffix array construction
Signature
namespace duckdb_zstd {
// --- Dictionary Training ---
// Train a dictionary from an array of concatenated samples.
// Returns size of dictionary stored into dictBuffer (<= dictBufferCapacity),
// or an error code testable with ZDICT_isError().
ZDICTLIB_API size_t ZDICT_trainFromBuffer(
void* dictBuffer, size_t dictBufferCapacity,
const void* samplesBuffer,
const size_t* samplesSizes, unsigned nbSamples);
// --- Dictionary Parameters ---
typedef struct {
int compressionLevel; // optimize for a specific compression level; 0 = default
unsigned notificationLevel; // log verbosity: 0=none, 1=errors, 2=progress, 3=details, 4=debug
unsigned dictID; // force dictionary ID; 0 = auto (random 32-bit value)
} ZDICT_params_t;
// --- Dictionary Finalization ---
// Add zstd header and entropy tables to raw dictionary content.
ZDICTLIB_API size_t ZDICT_finalizeDictionary(
void* dstDictBuffer, size_t maxDictSize,
const void* dictContent, size_t dictContentSize,
const void* samplesBuffer, const size_t* samplesSizes,
unsigned nbSamples, ZDICT_params_t parameters);
// --- Helper Functions ---
ZDICTLIB_API unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize);
ZDICTLIB_API size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize);
ZDICTLIB_API unsigned ZDICT_isError(size_t errorCode);
ZDICTLIB_API const char* ZDICT_getErrorName(size_t errorCode);
} // namespace duckdb_zstd
Import
#include "zdict.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| samplesBuffer | const void* |
Yes | Concatenated sample data for training; recommended ~100x target dictionary size |
| samplesSizes | const size_t* |
Yes | Array of byte-lengths for each sample in samplesBuffer
|
| nbSamples | unsigned |
Yes | Number of samples; a few thousand recommended |
| dictBuffer / dstDictBuffer | void* |
Yes | Pre-allocated buffer for the output dictionary |
| dictBufferCapacity / maxDictSize | size_t |
Yes | Maximum dictionary size; ~100 KB is typical |
| dictContent | const void* |
No | Raw content bytes for ZDICT_finalizeDictionary()
|
| parameters | ZDICT_params_t |
No | Training parameters: compression level, verbosity, dictionary ID |
Outputs
| Name | Type | Description |
|---|---|---|
| (return) | size_t |
Size of the trained dictionary written to the output buffer, or an error code testable via ZDICT_isError()
|
| dictBuffer | void* |
Contains the trained zstd dictionary (header + content) |
| dictID | unsigned |
Dictionary identifier extracted via ZDICT_getDictID()
|
Usage Examples
#include "zdict.h"
#include "zstd.h"
using namespace duckdb_zstd;
// --- Train a Dictionary ---
// Assume 'samples' is a buffer of concatenated sample data,
// and 'sampleSizes' is an array of individual sample sizes.
const size_t dictCapacity = 110 * 1024; // 110 KB
void* dictBuffer = malloc(dictCapacity);
size_t dictSize = ZDICT_trainFromBuffer(
dictBuffer, dictCapacity,
samples, sampleSizes, numSamples);
if (ZDICT_isError(dictSize)) {
fprintf(stderr, "Training failed: %s\n",
ZDICT_getErrorName(dictSize));
}
// --- Inspect the Dictionary ---
unsigned id = ZDICT_getDictID(dictBuffer, dictSize);
// --- Use the Dictionary for Compression ---
// Load via ZSTD_CCtx_loadDictionary(cctx, dictBuffer, dictSize);
// Load via ZSTD_DCtx_loadDictionary(dctx, dictBuffer, dictSize);
free(dictBuffer);