Implementation:Apache Flink BulkDecodingFormat
| Knowledge Sources | |
|---|---|
| Domains | Connectors, File_Connector |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A decoding format interface that extends DecodingFormat to produce BulkFormat readers for file-based dynamic table sources, with optional filter push-down support.
Description
BulkDecodingFormat is an internal generic interface that extends DecodingFormat<BulkFormat<T, FileSourceSplit>>. It bridges the gap between Flink's table connector format abstraction (DecodingFormat) and the file source's bulk reading API (BulkFormat) by binding the split type to FileSourceSplit.
The interface adds a single default method, applyFilters, which accepts a list of ResolvedExpression filters in conjunctive form. This enables format implementations to perform filter push-down on a best-effort basis, meaning formats may choose to apply some, all, or none of the provided filters depending on their capabilities. The default implementation is a no-op, making filter support entirely optional for format implementors.
This interface is the return type of BulkReaderFormatFactory.createDecodingFormat() and is the key abstraction that format plugins (such as Parquet, ORC, or Avro) implement to provide reading capabilities to the file system connector's table source.
Usage
Implement BulkDecodingFormat when creating a new file format that should integrate with Flink's file system connector as a table source. Override applyFilters to support filter push-down if the format supports predicate evaluation at the reader level (e.g., Parquet and ORC column predicate push-down).
Code Reference
Source Location
- Repository: Apache_Flink
- File: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/format/BulkDecodingFormat.java
- Lines: 1-37
Signature
@Internal
public interface BulkDecodingFormat<T> extends DecodingFormat<BulkFormat<T, FileSourceSplit>> {
default void applyFilters(List<ResolvedExpression> filters) {}
}
Import
import org.apache.flink.connector.file.table.format.BulkDecodingFormat;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| filters | List<ResolvedExpression> | No | A list of filter expressions in conjunctive form to push down to the format reader on a best-effort basis. |
Outputs
| Name | Type | Description |
|---|---|---|
| (inherited) createRuntimeDecoder result | BulkFormat<T, FileSourceSplit> | A bulk format reader capable of reading records of type T from file source splits (inherited from DecodingFormat). |
Usage Examples
// Implement a BulkDecodingFormat with filter push-down support
public class MyBulkDecodingFormat implements BulkDecodingFormat<RowData> {
private List<ResolvedExpression> filters = Collections.emptyList();
@Override
public BulkFormat<RowData, FileSourceSplit> createRuntimeDecoder(
DynamicTableSource.Context context,
DataType producedDataType) {
// Create a BulkFormat reader, optionally applying pushed-down filters
return new MyBulkFormat(producedDataType, filters);
}
@Override
public DataType getProducedDataType() {
return producedType;
}
@Override
public void applyFilters(List<ResolvedExpression> filters) {
// Store filters for push-down during reader creation
this.filters = filters;
}
@Override
public ChangelogMode getChangelogMode() {
return ChangelogMode.insertOnly();
}
}