Implementation:Apache Paimon ManifestListManager
| Knowledge Sources | |
|---|---|
| Domains | Manifest Management, Snapshot Management |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
ManifestListManager manages manifest list files in Avro format that contain metadata about manifest files associated with snapshots.
Description
The ManifestListManager class provides functionality for reading and writing manifest list files in Apache Paimon tables. Manifest lists are higher-level metadata files that point to individual manifest files and contain aggregate statistics about partitions.
Each manifest list file tracks multiple manifest files with their associated metadata including file sizes, counts of added/deleted files, partition statistics, schema IDs, and row ID ranges. The manager handles both base and delta manifest lists associated with snapshots.
The implementation uses Avro format for efficient serialization with support for binary row encoding of partition statistics. It provides methods to read all manifest files from a snapshot (both base and delta), read only delta manifests, or read individual manifest list files.
Usage
Use ManifestListManager when you need to access manifest file metadata from snapshots, implement snapshot readers, or manage the manifest list lifecycle in Apache Paimon tables.
Code Reference
Source Location
- Repository: Apache_Paimon
- File: paimon-python/pypaimon/manifest/manifest_list_manager.py
Signature
class ManifestListManager:
"""Manager for manifest list files in Avro format using unified FileIO."""
def __init__(self, table):
"""Initialize with a FileStoreTable instance."""
def read_all(self, snapshot: Optional[Snapshot]) -> List[ManifestFileMeta]:
"""Read all manifest files (base + delta) from a snapshot."""
def read_delta(self, snapshot: Snapshot) -> List[ManifestFileMeta]:
"""Read only delta manifest files from a snapshot."""
def read(self, manifest_list_name: str) -> List[ManifestFileMeta]:
"""Read manifest file metadata from a manifest list file."""
def write(self, file_name, manifest_file_metas: List[ManifestFileMeta]):
"""Write manifest file metadata to a manifest list file."""
Import
from pypaimon.manifest.manifest_list_manager import ManifestListManager
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| table | FileStoreTable | Yes | The file store table instance |
| snapshot | Snapshot | Yes (for read_all/read_delta) | Snapshot containing manifest list references |
| manifest_list_name | str | Yes (for read) | Name of manifest list file to read |
| manifest_file_metas | List[ManifestFileMeta] | Yes (for write) | Manifest file metadata to write |
Outputs
| Name | Type | Description |
|---|---|---|
| manifest_files | List[ManifestFileMeta] | List of manifest file metadata with statistics |
Usage Examples
from pypaimon.manifest.manifest_list_manager import ManifestListManager
# Create manifest list manager
list_manager = ManifestListManager(table)
# Read all manifest files from a snapshot
snapshot = table.snapshot_manager.get_latest_snapshot()
all_manifests = list_manager.read_all(snapshot)
# Read only delta manifests
delta_manifests = list_manager.read_delta(snapshot)
# Read a specific manifest list file
manifest_metas = list_manager.read("manifest-list-abc123")
# Process manifest file metadata
for meta in all_manifests:
print(f"Manifest: {meta.file_name}")
print(f"Size: {meta.file_size}")
print(f"Added files: {meta.num_added_files}")
print(f"Deleted files: {meta.num_deleted_files}")
print(f"Schema ID: {meta.schema_id}")
print(f"Row range: {meta.min_row_id} - {meta.max_row_id}")
# Write manifest list
new_metas = [meta1, meta2, meta3]
list_manager.write("manifest-list-new456", new_metas)