Jump to content

Connect SuperML | Leeroopedia MCP: Equip your AI agents with best practices, code verification, and debugging knowledge. Powered by Leeroo — building Organizational Superintelligence. Contact us at founders@leeroo.com.

Implementation:Duckdb Duckdb Zstd Common

From Leeroopedia


Knowledge Sources
Domains Compression, Third_Party
Last Updated 2026-02-07 12:00 GMT

Overview

Zstd Common contains the shared utility libraries, entropy coding primitives, hashing, threading infrastructure, and public header definitions used by both the Zstd compressor and decompressor within DuckDB.

Description

The common layer of the Zstd library in DuckDB (namespace duckdb_zstd) provides foundational components shared across all Zstd subsystems:

  • zstd.h (3093 lines) -- the master public API header defining the Simple API, Explicit Context API, Streaming API, constants (magic numbers, block sizes, version information), and error handling. Zstd version 1.5.6 with compression levels from ZSTD_minCLevel() (negative) through ZSTD_maxCLevel() (currently 22), default level 3.
  • fse.h (637 lines) -- Finite State Entropy (FSE) codec public prototypes (version 0.9.0). FSE is used for entropy coding of match lengths, literal lengths, and offsets. Provides FSE_compressBound(), error management, and detailed compression/decompression pipelines in 5-step and 3-step processes respectively.
  • compiler.h (369 lines) -- compiler abstraction macros for portability across GCC, Clang, MSVC, and others; includes force-inline, target attribute, and sanitizer annotations.
  • entropy_common.cpp (343 lines) -- shared entropy coding functions including FSE_readNCount() for reading normalized frequency counts and version/error management for both FSE and Huffman coders.
  • fse_decompress.cpp (316 lines) -- FSE table decompression used during both zstd decompression and dictionary loading.
  • pool.cpp (376 lines) -- thread pool implementation with job queue, mutex-based synchronization, and configurable thread count. Used by the multi-threaded compressor (zstdmt_compress) and COVER dictionary trainer.
  • xxhash.cpp (863 lines) -- xxHash fast hashing (XXH32/XXH64) used for content checksums, dictionary ID hashing, and internal hash tables. Supports configurable memory access strategies (memcpy, packed, direct) for different architectures.

Usage

These common modules are compiled into every DuckDB build that includes Zstd support. They are not called directly by DuckDB application code but are linked by the Zstd compressor, decompressor, and dictionary modules. The zstd.h header is the primary include file for all Zstd functionality.

Code Reference

Source Location

Signature

namespace duckdb_zstd {

// --- zstd.h: Version and Constants ---
#define ZSTD_VERSION_MAJOR    1
#define ZSTD_VERSION_MINOR    5
#define ZSTD_VERSION_RELEASE  6
#define ZSTD_CLEVEL_DEFAULT   3
#define ZSTD_MAGICNUMBER      0xFD2FB528
#define ZSTD_MAGIC_DICTIONARY 0xEC30A437

ZSTDLIB_API unsigned    ZSTD_versionNumber(void);
ZSTDLIB_API const char* ZSTD_versionString(void);
ZSTDLIB_API unsigned    ZSTD_isError(size_t code);
ZSTDLIB_API const char* ZSTD_getErrorName(size_t code);

// --- fse.h: Finite State Entropy ---
#define FSE_VERSION_MAJOR   0
#define FSE_VERSION_MINOR   9
#define FSE_VERSION_RELEASE 0

FSE_PUBLIC_API size_t      FSE_compressBound(size_t size);
FSE_PUBLIC_API unsigned    FSE_isError(size_t code);
FSE_PUBLIC_API const char* FSE_getErrorName(size_t code);
FSE_PUBLIC_API unsigned    FSE_versionNumber(void);

// --- entropy_common.cpp ---
unsigned FSE_isError(size_t code);
unsigned HUF_isError(size_t code);
const char* FSE_getErrorName(size_t code);
const char* HUF_getErrorName(size_t code);
size_t FSE_readNCount(short* normalizedCounter, unsigned* maxSVPtr,
                      unsigned* tableLogPtr,
                      const void* headerBuffer, size_t hbSize);

} // namespace duckdb_zstd

Import

#include "zstd.h"              // master API
#include "zstd/common/fse.h"   // FSE codec
#include "zstd/common/huf.h"   // Huffman codec

I/O Contract

Inputs

Name Type Required Description
code size_t Yes Return value from a Zstd/FSE/HUF function to test for errors
headerBuffer const void* Yes (for FSE_readNCount) Buffer containing serialized normalized frequency counts
hbSize size_t Yes (for FSE_readNCount) Size of the header buffer; must be >= 8 bytes for the optimized path
size size_t Yes (for FSE_compressBound) Uncompressed input size for computing maximum FSE-compressed output size

Outputs

Name Type Description
(return) from ZSTD_isError unsigned 1 if the code represents an error, 0 otherwise
(return) from ZSTD_getErrorName const char* Human-readable error description string
(return) from FSE_compressBound size_t Maximum compressed size for FSE encoding
normalizedCounter (out param) short* Decoded frequency table from FSE_readNCount()
maxSVPtr (out param) unsigned* Maximum symbol value found in the table
tableLogPtr (out param) unsigned* Log2 of the table size

Usage Examples

#include "zstd.h"

using namespace duckdb_zstd;

// --- Version Check ---
unsigned version = ZSTD_versionNumber();  // e.g., 10506
const char* vstr = ZSTD_versionString();  // e.g., "1.5.6"

// --- Error Handling Pattern ---
size_t result = ZSTD_compress(dst, dstCap, src, srcSize, 3);
if (ZSTD_isError(result)) {
    const char* errName = ZSTD_getErrorName(result);
    // Handle error: errName is a readable string like "Src size is incorrect"
}

// --- Compression Level Bounds ---
int minLevel = ZSTD_minCLevel();   // negative value for fastest
int maxLevel = ZSTD_maxCLevel();   // 22 (ultra)
int defLevel = ZSTD_defaultCLevel(); // 3

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment
Retrieved from "https://leeroopedia.com/index.php?title=Implementation:Duckdb_Duckdb_Zstd_Common&oldid=12939"