Implementation:Apache Paimon ManifestFileManager
| Knowledge Sources | |
|---|---|
| Domains | Manifest Management, File I/O |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
ManifestFileManager is a writer and reader for manifest files in Avro format that manages the serialization and deserialization of manifest entries.
Description
The ManifestFileManager class provides comprehensive functionality for reading and writing manifest files in Apache Paimon tables. It handles both single-threaded and parallel reading of manifest entries with support for filtering and statistics optimization.
Manifest files contain metadata about data files including file locations, row counts, key/value statistics, sequence numbers, and schema information. The manager serializes and deserializes these entries using Avro format with a unified FileIO interface for cross-platform compatibility.
The implementation supports parallel reading with configurable worker threads, automatic deduplication of ADD and DELETE entries, and optional statistics dropping to reduce memory overhead. It properly handles timestamp conversions, binary row serialization, and schema evolution through value statistics field mapping.
Usage
Use ManifestFileManager when you need to read or write manifest files in Apache Paimon, particularly for snapshot management, data file tracking, or implementing custom table operations that require manifest metadata access.
Code Reference
Source Location
- Repository: Apache_Paimon
- File: paimon-python/pypaimon/manifest/manifest_file_manager.py
Signature
class ManifestFileManager:
"""Writer for manifest files in Avro format using unified FileIO."""
def __init__(self, table):
"""Initialize with a FileStoreTable instance."""
def read_entries_parallel(
self,
manifest_files: List[ManifestFileMeta],
manifest_entry_filter=None,
drop_stats=True,
max_workers=8
) -> List[ManifestEntry]:
"""Read manifest entries in parallel from multiple manifest files."""
def read(
self,
manifest_file_name: str,
manifest_entry_filter=None,
drop_stats=True
) -> List[ManifestEntry]:
"""Read manifest entries from a single manifest file."""
def write(self, file_name, entries: List[ManifestEntry]):
"""Write manifest entries to a manifest file."""
def _get_value_stats_fields(self, file_dict: dict, schema_fields: list) -> List:
"""Get value stats fields from file metadata."""
Import
from pypaimon.manifest.manifest_file_manager import ManifestFileManager
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| table | FileStoreTable | Yes | The file store table instance |
| manifest_files | List[ManifestFileMeta] | Yes (for parallel read) | List of manifest file metadata to read |
| manifest_file_name | str | Yes (for single read) | Name of manifest file to read |
| manifest_entry_filter | Callable | No | Optional filter function for entries |
| drop_stats | bool | No | Whether to drop statistics (default True) |
| max_workers | int | No | Maximum parallel workers (default 8) |
| entries | List[ManifestEntry] | Yes (for write) | Manifest entries to write |
Outputs
| Name | Type | Description |
|---|---|---|
| entries | List[ManifestEntry] | List of manifest entries with file metadata |
Usage Examples
from pypaimon.manifest.manifest_file_manager import ManifestFileManager
# Create manifest file manager
manifest_manager = ManifestFileManager(table)
# Read manifest entries from a single file
entries = manifest_manager.read("manifest-abc123", drop_stats=True)
# Read entries in parallel from multiple files
manifest_files = [meta1, meta2, meta3]
all_entries = manifest_manager.read_entries_parallel(
manifest_files,
max_workers=4
)
# Filter entries during read
def my_filter(entry):
return entry.bucket == 0
filtered_entries = manifest_manager.read(
"manifest-def456",
manifest_entry_filter=my_filter
)
# Write manifest entries
new_entries = [entry1, entry2, entry3]
manifest_manager.write("manifest-new789", new_entries)
# Process entries
for entry in entries:
print(f"File: {entry.file.file_name}")
print(f"Rows: {entry.file.row_count}")
print(f"Partition: {entry.partition}")