Principle:Apache Paimon Distributed Write via Ray
| Knowledge Sources | |
|---|---|
| Domains | Data_Lake, Data_Ingestion |
| Last Updated | 2026-02-07 00:00 GMT |
Overview
Mechanism for writing distributed Ray Datasets to Paimon tables with parallel execution and atomic commit.
Description
Distributed write via Ray enables parallel data ingestion from a Ray Dataset into a Paimon table. The write_ray() method creates a PaimonDatasink that distributes write tasks across Ray workers. Each worker writes its assigned data blocks to the file store, and the datasink's on_write_complete() callback handles the atomic commit to make all data visible simultaneously.
The write process follows these stages:
- Datasink creation: write_ray() instantiates a PaimonDatasink configured with the table's write builder and commit parameters.
- Task distribution: Ray Data's write_datasink() distributes data blocks across available workers based on the concurrency parameter.
- Local writes: Each Ray write task creates a local FileStoreWrite instance, writes its assigned data blocks to the file store, and produces CommitMessage objects describing the written files.
- Commit message collection: All CommitMessage objects are returned to the driver node.
- Atomic commit: The on_write_complete() callback on the driver collects all commit messages and performs a single atomic commit via FileStoreCommit, making all written data visible in one snapshot.
This architecture combines Ray's distributed execution model with Paimon's ACID commit protocol for scalable, consistent data ingestion.
Usage
Use when ingesting large volumes of data from distributed sources into Paimon tables where parallel writing provides significant performance benefits. This is the core write step in the Ray-based Paimon ingestion pipeline, executed after data loading and schema alignment.
Theoretical Basis
This principle follows the distributed write-then-commit pattern. Write tasks are parallelized across workers (map phase), and the commit is centralized on the driver (reduce phase).
Key theoretical properties:
- Atomicity: All data from a single write_ray() call becomes visible in a single snapshot. Readers either see all the data or none of it, never a partial write. This is achieved by deferring the commit until all write tasks complete.
- Consistency: The centralized commit on the driver ensures that the snapshot metadata is consistent with the files written by all workers. The commit protocol validates that all expected commit messages are present before finalizing.
- Isolation: Concurrent readers are not affected by in-progress writes. They continue reading from the previous snapshot until the new snapshot is committed.
- Durability: Once the commit succeeds, the data is persisted in the file store and the snapshot is recorded in the catalog. The data survives process failures after commit.
The PaimonDatasink implements Ray's Datasink interface, which provides the contract for plugging custom write targets into Ray Data's write pipeline. This interface defines:
- write() -- called on each worker to write a data block.
- on_write_complete() -- called on the driver after all write tasks finish.