Implementation:Duckdb Duckdb Zstd Decompressor

Knowledge Sources	Duckdb_Duckdb Zstd
Domains	Compression, Third_Party
Last Updated	2026-02-07 12:00 GMT

Overview

Zstd Decompressor is the decompression side of the Zstandard library integrated into DuckDB, responsible for decoding zstd-compressed frames back into original data at high throughput.

Description

The Zstd decompressor in DuckDB consists of three main modules within the duckdb_zstd namespace:

• zstd_decompress.cpp (2411 lines) -- top-level decompression entry points including ZSTD_decompress(), ZSTD_decompressDCtx(), decompression context management, DDict hashset for multi-dictionary lookups, and streaming decompression via ZSTD_decompressStream()
• zstd_decompress_block.cpp (2218 lines) -- block-level decompression that handles literal decoding, sequence decoding, and match copying within individual zstd blocks
• huf_decompress.cpp (1912 lines) -- Huffman decoder with both single-stream (X1) and double-stream (X2) variants, featuring optional BMI2 acceleration for fast decoding

Key configuration:

• Heap-mode by default (ZSTD_HEAPMODE = 1) for decompression context allocation
• Maximum window size defaults to (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) + 1, configurable via ZSTD_DCtx_setMaxWindowSize()
• No-forward-progress limit of 16 calls to ZSTD_decompressStream() before triggering an error
• DDict hashset for efficient multi-dictionary decompression with 0.75 load factor and power-of-2 table sizes

Usage

DuckDB uses the Zstd decompressor to read zstd-compressed data blocks from its storage layer and to decode Parquet files compressed with the Zstd codec. The decompression context (ZSTD_DCtx) is reusable across multiple decompression operations for reduced allocation overhead.

Code Reference

Source Location

• Repository: Duckdb_Duckdb
• Files:
  • third_party/zstd/decompress/zstd_decompress.cpp (2411 lines) -- main decompressor
  • third_party/zstd/decompress/zstd_decompress_block.cpp (2218 lines) -- block-level decompression
  • third_party/zstd/decompress/huf_decompress.cpp (1912 lines) -- Huffman decoder

Signature

namespace duckdb_zstd {

// --- Simple Decompression API ---

// Decompress a zstd frame from src into dst.
// Returns the number of decompressed bytes, or an error code.
ZSTDLIB_API size_t ZSTD_decompress(void* dst, size_t dstCapacity,
                                   const void* src, size_t compressedSize);

// --- Frame Inspection ---

// Get the decompressed content size from the frame header.
// Returns ZSTD_CONTENTSIZE_UNKNOWN or ZSTD_CONTENTSIZE_ERROR on failure.
ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(
    const void* src, size_t srcSize);

// Find the compressed size of the first frame.
ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(
    const void* src, size_t srcSize);

// --- Explicit Context API ---

typedef struct ZSTD_DCtx_s ZSTD_DCtx;
ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx(void);
ZSTDLIB_API size_t      ZSTD_freeDCtx(ZSTD_DCtx* dctx);

// Decompress using an explicit context (reusable, supports sticky parameters).
ZSTDLIB_API size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
                                       void* dst, size_t dstCapacity,
                                       const void* src, size_t srcSize);

// --- Error Handling ---
ZSTDLIB_API unsigned    ZSTD_isError(size_t code);
ZSTDLIB_API const char* ZSTD_getErrorName(size_t code);

} // namespace duckdb_zstd

Import

#include "zstd.h"

I/O Contract

Inputs

Name	Type	Required	Description
src	const void*	Yes	Pointer to zstd-compressed frame data
compressedSize / srcSize	size_t	Yes	Exact size of the compressed frame(s); must be exact for ZSTD_decompress()
dst	void*	Yes	Pointer to pre-allocated destination buffer for decompressed data
dstCapacity	size_t	Yes	Upper bound on the decompressed size; the buffer must be at least this large
dctx	ZSTD_DCtx*	No	Reusable decompression context for ZSTD_decompressDCtx()

Outputs

Name	Type	Description
(return) from ZSTD_decompress	size_t	Number of decompressed bytes written to dst, or an error code testable via ZSTD_isError()
(return) from ZSTD_getFrameContentSize	unsigned long long	Decompressed content size, ZSTD_CONTENTSIZE_UNKNOWN, or ZSTD_CONTENTSIZE_ERROR
dst buffer	void*	Contains the decompressed output data

Usage Examples

#include "zstd.h"

using namespace duckdb_zstd;

// Assuming 'compressed' and 'compressedSize' are available from a prior
// ZSTD_compress() call.

// --- Query frame content size ---
unsigned long long decompSize = ZSTD_getFrameContentSize(
    compressed, compressedSize);
if (decompSize == ZSTD_CONTENTSIZE_ERROR) {
    // Not a valid zstd frame
}

// --- One-shot Decompression ---
void* decompressed = malloc(decompSize);
size_t result = ZSTD_decompress(decompressed, decompSize,
                                compressed, compressedSize);
if (ZSTD_isError(result)) {
    fprintf(stderr, "Decompression error: %s\n",
            ZSTD_getErrorName(result));
}

// --- Context-based Decompression (reusable) ---
ZSTD_DCtx* dctx = ZSTD_createDCtx();
result = ZSTD_decompressDCtx(dctx, decompressed, decompSize,
                             compressed, compressedSize);
ZSTD_freeDCtx(dctx);
free(decompressed);

Related Pages

• Principle:Duckdb_Duckdb_Zstandard_Compression