Implementation:Apache Paimon SstFileReader
| Knowledge Sources | |
|---|---|
| Domains | Global Index, B-Tree |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
SstFileReader provides point and range query capabilities for B-tree index SST files with compression and checksum verification.
Description
SstFileReader is the main entry point for reading B-tree global index files (SST files). It implements both point queries (looking up a specific key) and range queries (iterating over a key range) by managing the two-level B-tree structure: an index block that maps key ranges to data blocks, and data blocks that contain the actual key-value entries.
During initialization, the reader loads the index block using the handle from the BTreeFileFooter. The index block contains entries where keys are the maximum keys in data blocks and values are BlockHandles pointing to those data blocks. This structure enables efficient binary search to locate the correct data block for any query key.
The reader handles block decompression and integrity verification. Each block is stored with a 5-byte trailer containing a compression type (1 byte) and CRC32 checksum (4 bytes). Currently supported compression types are NONE (0) and ZSTD (1). For compressed blocks, the first few bytes contain a variable-length integer indicating the expected decompressed size, followed by the compressed data.
Range queries are supported through SstFileIterator, which maintains state as it scans through data blocks. The iterator uses the index block to navigate between data blocks and uses BlockIterator for scanning within blocks. The seek_to method enables starting iteration from any key position using binary search.
Usage
SstFileReader is used by the global index lookup system to query B-tree index files. Applications create a reader with a file handle and comparator, then use create_iterator for range scans or implement point queries on top of the iteration interface.
Code Reference
Source Location
- Repository: Apache_Paimon
- File: paimon-python/pypaimon/globalindex/btree/sst_file_reader.py
Signature
class SstFileIterator:
"""
Iterator for range queries on SST file.
"""
def __init__(self, read_block: Callable[[BlockHandle], BlockReader],
index_block_iterator: BlockIterator):
...
def seek_to(self, key: bytes) -> None:
"""Seek to the position of record with key >= specified key."""
...
def read_batch(self) -> Optional[BlockIterator]:
"""Read a batch of records and advance to next batch."""
...
class SstFileReader:
"""
An SST File Reader which serves point queries and range queries.
Note that this class is NOT thread-safe.
"""
def __init__(
self,
input_stream: BinaryIO,
comparator: Callable[[bytes, bytes], int],
index_block_handle: BlockHandle
):
...
def _read_block(self, block_handle: BlockHandle) -> BlockReader:
"""Read and decompress block with checksum verification."""
...
def create_iterator(self) -> SstFileIterator:
"""Create iterator for range queries."""
...
@staticmethod
def crc32c(bytes_data: bytes, compression_type_id: int) -> int:
"""Calculate CRC32 checksum."""
...
def close(self) -> None:
"""Close the reader and release resources."""
...
Import
from pypaimon.globalindex.btree.sst_file_reader import SstFileReader, SstFileIterator
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| input_stream | BinaryIO | Yes | File handle for SST file |
| comparator | Callable[[bytes, bytes], int] | Yes | Key comparison function |
| index_block_handle | BlockHandle | Yes | Handle to index block from footer |
Outputs
| Name | Type | Description |
|---|---|---|
| SstFileIterator | SstFileIterator | Iterator for range queries |
| BlockIterator | Optional[BlockIterator] | Iterator over entries in current batch |
| BlockEntry | BlockEntry | Key-value entry from index |
Usage Examples
from pypaimon.globalindex.btree.sst_file_reader import SstFileReader
from pypaimon.globalindex.btree.btree_file_footer import BTreeFileFooter
# Open SST file
with open("index.sst", "rb") as f:
# Read footer
f.seek(-BTreeFileFooter.ENCODED_LENGTH, 2)
footer_bytes = f.read(BTreeFileFooter.ENCODED_LENGTH)
footer = BTreeFileFooter.read_footer(footer_bytes)
# Create reader with key comparator
def key_comparator(key1: bytes, key2: bytes) -> int:
# Compare as strings
s1, s2 = key1.decode(), key2.decode()
return (s1 > s2) - (s1 < s2)
f.seek(0) # Reset to beginning
reader = SstFileReader(
input_stream=f,
comparator=key_comparator,
index_block_handle=footer.index_block_handle
)
# Range query: iterate from "alice" onward
iterator = reader.create_iterator()
iterator.seek_to(b"alice")
while True:
batch = iterator.read_batch()
if batch is None:
break
# Process entries in batch
while batch.has_next():
entry = next(batch)
key = entry.key.decode()
value = entry.value.decode()
print(f"{key} -> {value}")
reader.close()
# Point query implementation
def point_query(reader: SstFileReader, target_key: bytes) -> Optional[bytes]:
"""Look up a single key."""
iterator = reader.create_iterator()
iterator.seek_to(target_key)
batch = iterator.read_batch()
if batch and batch.has_next():
entry = next(batch)
if reader.comparator(entry.key, target_key) == 0:
return entry.value
return None
result = point_query(reader, b"alice")
if result:
print(f"Found: {result.decode()}")