Implementation:Apache Flink Mapred HadoopOutputFormat
| Knowledge Sources | |
|---|---|
| Domains | Connectors, Hadoop_Compatibility |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A Flink wrapper that adapts Hadoop's legacy mapred OutputFormat API so that Flink can write key-value Tuple2<K, V> records through any Hadoop mapred-compatible output format.
Description
HadoopOutputFormat is a concrete implementation in the org.apache.flink.api.java.hadoop.mapred package that extends HadoopOutputFormatBase<K, V, Tuple2<K, V>>. It provides the final bridge from Flink's output pipeline to the Hadoop mapred OutputFormat, handling the extraction of key and value fields from Tuple2 records and delegating the actual writing to the Hadoop RecordWriter.
The class provides two constructors: one that accepts a Hadoop OutputFormat<K, V> and a JobConf, and another that additionally accepts an OutputCommitter class which is then set on the JobConf. The writeRecord method extracts record.f0 (key) and record.f1 (value) from the incoming tuple and passes them directly to the underlying Hadoop RecordWriter.
This class is annotated with @Public, indicating it is part of Flink's stable public API.
Usage
Use this class when you need to write data to Hadoop-compatible data sinks using the legacy org.apache.hadoop.mapred.OutputFormat API within a Flink batch program. This is useful for writing to HDFS, HBase, or other Hadoop-ecosystem storage destinations using existing Hadoop OutputFormat implementations.
Code Reference
Source Location
- Repository: Apache_Flink
- File: flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormat.java
- Lines: 1-60
Signature
@Public
public class HadoopOutputFormat<K, V> extends HadoopOutputFormatBase<K, V, Tuple2<K, V>>
Import
import org.apache.flink.api.java.hadoop.mapred.HadoopOutputFormat;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| mapredOutputFormat | org.apache.hadoop.mapred.OutputFormat<K, V> | Yes | The Hadoop mapred OutputFormat instance to wrap |
| job | org.apache.hadoop.mapred.JobConf | Yes | The Hadoop JobConf for configuration |
| outputCommitterClass | Class<OutputCommitter> | No | Optional OutputCommitter class to set on the JobConf |
Outputs
| Name | Type | Description |
|---|---|---|
| record | Tuple2<K, V> | The key-value tuple to be written; f0 is written as the key and f1 as the value through the Hadoop RecordWriter |
Usage Examples
// Create a HadoopOutputFormat wrapping a Hadoop TextOutputFormat
import org.apache.flink.api.java.hadoop.mapred.HadoopOutputFormat;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.mapred.JobConf;
JobConf jobConf = new JobConf();
jobConf.set("mapred.output.dir", "/path/to/output");
HadoopOutputFormat<LongWritable, Text> hadoopOF =
new HadoopOutputFormat<>(new TextOutputFormat<LongWritable, Text>(), jobConf);
// Use hadoopOF as a Flink OutputFormat
DataSet<Tuple2<LongWritable, Text>> data = ...;
data.output(hadoopOF);