Implementation:Apache Flink PartitionTimeExtractor
| Knowledge Sources | |
|---|---|
| Domains | Connectors, File_Connector |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A serializable interface for extracting a LocalDateTime from partition key-value pairs, used to determine partition time for commit trigger evaluation.
Description
PartitionTimeExtractor is an experimental, serializable interface that extracts a LocalDateTime from a list of partition keys and their corresponding values. This extracted time is typically used by partition commit triggers to determine when a partition is ready to be committed (e.g., based on watermark progress or processing time).
The interface defines two extractor kinds as constants:
- DEFAULT ("default") - Uses the built-in DefaultPartTimeExtractor which parses partition values according to configurable extractor and formatter patterns.
- CUSTOM ("custom") - Loads a user-provided class implementing PartitionTimeExtractor via reflection using the specified class loader.
The static create factory method handles instantiation of both kinds. For the default kind, it creates a DefaultPartTimeExtractor with the given extractor pattern and formatter pattern. For the custom kind, it dynamically loads and instantiates the specified class using Class.newInstance() via the provided class loader, throwing a RuntimeException if the class cannot be found or instantiated.
Usage
Use PartitionTimeExtractor to configure how partition time is determined for time-based partition commit triggers in file-based table sinks. Use the default extractor with patterns for standard date/time partition layouts, or implement a custom extractor class for non-standard partition time semantics. The extractor is typically configured through table options for the file system connector.
Code Reference
Source Location
- Repository: Apache_Flink
- File: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/PartitionTimeExtractor.java
- Lines: 1-61
Signature
@Experimental
public interface PartitionTimeExtractor extends Serializable {
String DEFAULT = "default";
String CUSTOM = "custom";
LocalDateTime extract(List<String> partitionKeys, List<String> partitionValues);
static PartitionTimeExtractor create(
ClassLoader userClassLoader,
String extractorKind,
String extractorClass,
String extractorPattern,
String formatterPattern);
}
Import
import org.apache.flink.connector.file.table.PartitionTimeExtractor;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| partitionKeys | List<String> | Yes | The names of the partition columns (e.g., ["year", "month", "day"]). |
| partitionValues | List<String> | Yes | The corresponding values of the partition columns (e.g., ["2024", "01", "15"]). |
| userClassLoader | ClassLoader | Yes (for factory) | Class loader used to load custom extractor classes. |
| extractorKind | String | Yes (for factory) | Either "default" or "custom", selecting the extractor type. |
| extractorClass | String | Conditional | Fully qualified class name of a custom PartitionTimeExtractor implementation; required when extractorKind is "custom". |
| extractorPattern | String | No | Pattern used by the default extractor to compose a timestamp string from partition values. |
| formatterPattern | String | No | Date-time formatter pattern used by the default extractor to parse the composed timestamp string. |
Outputs
| Name | Type | Description |
|---|---|---|
| extract() result | LocalDateTime | The extracted partition time derived from the partition keys and values. |
Usage Examples
// Create a default partition time extractor with year/month/day pattern
PartitionTimeExtractor extractor = PartitionTimeExtractor.create(
Thread.currentThread().getContextClassLoader(),
"default",
null, // no custom class needed
"$year-$month-$day", // extractor pattern
"yyyy-MM-dd" // formatter pattern
);
// Extract time from partition values
List<String> keys = Arrays.asList("year", "month", "day");
List<String> values = Arrays.asList("2024", "01", "15");
LocalDateTime partitionTime = extractor.extract(keys, values);
// Result: 2024-01-15T00:00
// Create a custom partition time extractor
PartitionTimeExtractor customExtractor = PartitionTimeExtractor.create(
userClassLoader,
"custom",
"com.example.MyPartitionTimeExtractor",
null,
null
);