Principle:Apache Flink File Enumeration and Splitting
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Distributed_Computing |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A file discovery and partitioning mechanism that recursively enumerates input paths and divides files into splits for parallel processing.
Description
File Enumeration and Splitting addresses the problem of converting a set of input paths into parallelizable work units (splits). The enumerator recursively traverses directories, filters files, and creates FileSourceSplit objects that define byte-range segments of files. Two strategies exist:
- Non-splitting: Creates one split per file (for non-splittable formats like compressed files)
- Block-splitting: Creates multiple splits per file based on HDFS block boundaries (for splittable formats), enabling finer-grained parallelism
Each split contains the file path, byte offset, length, and optional host locality hints from the distributed filesystem.
Usage
This principle operates automatically based on the formats splittability. Block splitting is preferred for large files on HDFS where data locality matters. Non-splitting is required for formats that cannot be read from arbitrary offsets (compressed files, some binary formats).
Theoretical Basis
// Abstract algorithm
function enumerateSplits(paths, minDesiredSplits):
splits = []
for each path in paths:
files = recursivelyListFiles(path, fileFilter)
for each file in files:
if file.isSplittable():
blocks = getBlockLocations(file)
for each block in blocks:
splits.add(new Split(file.path, block.offset, block.length, block.hosts))
else:
splits.add(new Split(file.path, 0, file.length, file.hosts))
return splits