Implementation:Apache Paimon BTreeFileFooter
| Knowledge Sources | |
|---|---|
| Domains | Global Index, B-Tree |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
BTreeFileFooter stores metadata about B-tree index file structure including handles to bloom filter, index block, and null bitmap.
Description
BTreeFileFooter represents the footer structure of a B-tree index file (SST file). It is stored at the end of the file and contains handles (offset and size information) to three key components: the bloom filter for fast negative lookups, the index block for navigating to data blocks, and the null bitmap for tracking null values.
The footer has a fixed encoding length of 48 bytes, ending with a 4-byte magic number (198732882) for file format validation. The structure includes three handle types: BloomFilterHandle (16 bytes: 8-byte offset, 4-byte size, 4-byte alignment, 8-byte expected entries), BlockHandle for index and null bitmap (12 bytes each: 8-byte offset, 4-byte size), and padding to reach the fixed size.
The bloom filter handle may be null (all zeros) if bloom filtering is not used. Similarly, the null bitmap handle may be null if the index doesn't contain null values. The index block handle is always present as it is essential for navigating the B-tree structure.
The footer is read from the last 48 bytes of the SST file. The magic number verification ensures that the file is a valid B-tree index file. Once read, the handles provide the information needed to load and navigate the other components of the index.
Usage
BTreeFileFooter is used by SstFileReader during initialization to locate and load the index block. Applications don't typically interact with the footer directly, but it is crucial for the internal operation of the global index reading system.
Code Reference
Source Location
- Repository: Apache_Paimon
- File: paimon-python/pypaimon/globalindex/btree/btree_file_footer.py
Signature
@dataclass
class BloomFilterHandle:
"""Handle for bloom filter block."""
offset: int
size: int
expected_entries: int
def is_null(self) -> bool:
"""Check if this handle represents a null bloom filter."""
...
class BTreeFileFooter:
"""
The Footer for BTree file.
"""
MAGIC_NUMBER = 198732882
ENCODED_LENGTH = 48
def __init__(
self,
bloom_filter_handle: Optional[BloomFilterHandle],
index_block_handle: BlockHandle,
null_bitmap_handle: Optional[BlockHandle]
):
...
@classmethod
def read_footer(cls, data: bytes) -> 'BTreeFileFooter':
"""Read footer from byte data."""
...
Import
from pypaimon.globalindex.btree.btree_file_footer import BTreeFileFooter, BloomFilterHandle
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| data | bytes | Yes | Footer bytes (48 bytes) |
Outputs
| Name | Type | Description |
|---|---|---|
| BTreeFileFooter | BTreeFileFooter | Parsed footer with handles |
| bloom_filter_handle | Optional[BloomFilterHandle] | Handle to bloom filter block |
| index_block_handle | BlockHandle | Handle to index block (always present) |
| null_bitmap_handle | Optional[BlockHandle] | Handle to null bitmap block |
Usage Examples
from pypaimon.globalindex.btree.btree_file_footer import BTreeFileFooter
# Open SST file
with open("index.sst", "rb") as f:
# Seek to footer position (last 48 bytes)
f.seek(-BTreeFileFooter.ENCODED_LENGTH, 2)
footer_bytes = f.read(BTreeFileFooter.ENCODED_LENGTH)
# Parse footer
try:
footer = BTreeFileFooter.read_footer(footer_bytes)
print(f"Magic number verified: {footer.MAGIC_NUMBER}")
# Check bloom filter
if footer.bloom_filter_handle and not footer.bloom_filter_handle.is_null():
print(f"Bloom filter at offset {footer.bloom_filter_handle.offset}, "
f"size {footer.bloom_filter_handle.size}")
# Index block is always present
print(f"Index block at offset {footer.index_block_handle.offset}, "
f"size {footer.index_block_handle.size}")
# Check null bitmap
if footer.null_bitmap_handle:
print(f"Null bitmap at offset {footer.null_bitmap_handle.offset}, "
f"size {footer.null_bitmap_handle.size}")
# Use handles to read blocks
f.seek(footer.index_block_handle.offset)
index_block_data = f.read(footer.index_block_handle.size)
except ValueError as e:
print(f"Invalid footer: {e}")