Implementation:Apache Paimon IndexManifestFile
| Knowledge Sources | |
|---|---|
| Domains | Manifest Management, Index Storage |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
IndexManifestFile is a reader for index manifest entries that supports both Avro and JSON formats for reading index metadata from manifest files.
Description
The IndexManifestFile class provides functionality to read index manifest files in Apache Paimon tables. It handles two file formats: Avro (used in production) and JSON (used in Python tests). The class automatically detects the file format based on the magic bytes at the beginning of the file.
Index manifest files contain metadata about index files including deletion vectors, file statistics, and global index information. The reader deserializes these entries and reconstructs IndexManifestEntry objects with complete metadata including partition information, bucket numbers, and index file details.
The implementation supports reading deletion vector metadata, which tracks deleted rows in data files, and global index metadata for efficient data lookups. Error handling is built in to provide clear diagnostics when files cannot be read or parsed correctly.
Usage
Use IndexManifestFile when you need to read index metadata from Apache Paimon tables, particularly for deletion vector tracking, index file management, or when implementing custom index readers.
Code Reference
Source Location
- Repository: Apache_Paimon
- File: paimon-python/pypaimon/manifest/index_manifest_file.py
Signature
class IndexManifestFile:
"""Index manifest file reader for reading index manifest entries."""
DELETION_VECTORS_INDEX = "DELETION_VECTORS"
def __init__(self, table):
"""Initialize with a FileStoreTable instance."""
def read(self, index_manifest_name: str) -> List[IndexManifestEntry]:
"""Read index manifest entries from the specified manifest file."""
def _read_avro(self, file_bytes: bytes, index_manifest_path: str) -> List[IndexManifestEntry]:
"""Read Avro formatted index manifest."""
def _read_json(self, file_bytes: bytes, index_manifest_path: str) -> List[IndexManifestEntry]:
"""Read JSON formatted index manifest (used by Python tests)."""
def _parse_global_index_meta(self, global_index_record) -> Optional[GlobalIndexMeta]:
"""Parse global index meta from Avro record."""
Import
from pypaimon.manifest.index_manifest_file import IndexManifestFile
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| table | FileStoreTable | Yes | The file store table instance containing table path and file IO |
| index_manifest_name | str | Yes | Name of the index manifest file to read |
Outputs
| Name | Type | Description |
|---|---|---|
| entries | List[IndexManifestEntry] | List of index manifest entries containing index file metadata |
Usage Examples
from pypaimon.manifest.index_manifest_file import IndexManifestFile
# Create index manifest file reader
index_reader = IndexManifestFile(table)
# Read index manifest entries
entries = index_reader.read("index-manifest-abc123")
# Process entries
for entry in entries:
print(f"Partition: {entry.partition}")
print(f"Bucket: {entry.bucket}")
print(f"Index type: {entry.index_file.index_type}")
print(f"File name: {entry.index_file.file_name}")
# Access deletion vector metadata if present
if entry.index_file.dv_ranges:
for file_name, dv_meta in entry.index_file.dv_ranges.items():
print(f"DV for {file_name}: offset={dv_meta.offset}, length={dv_meta.length}")