Implementation:Apache Flink Mapreduce HadoopOutputFormat
| Knowledge Sources | |
|---|---|
| Domains | Connectors, Hadoop_Compatibility |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A Flink wrapper that adapts Hadoop's new mapreduce OutputFormat API so that Flink can write key-value Tuple2<K, V> records through any Hadoop mapreduce-compatible output format.
Description
HadoopOutputFormat is a concrete implementation in the org.apache.flink.api.java.hadoop.mapreduce package that extends HadoopOutputFormatBase<K, V, Tuple2<K, V>>. It provides the final bridge from Flink's output pipeline to the new-style Hadoop mapreduce OutputFormat, handling the extraction of key and value fields from Tuple2 records and delegating the actual writing to the Hadoop RecordWriter.
The class provides a single constructor that accepts a Hadoop OutputFormat<K, V> and a Job instance. The writeRecord method extracts record.f0 (key) and record.f1 (value) from the incoming tuple and passes them to the underlying Hadoop RecordWriter.write(). Unlike the mapred variant, this version wraps InterruptedException in IOException since the new Hadoop API's RecordWriter.write() method declares this checked exception.
This class is annotated with @Public, indicating it is part of Flink's stable public API.
Usage
Use this class when you need to write data to Hadoop-compatible data sinks using the new org.apache.hadoop.mapreduce.OutputFormat API within a Flink batch program. This is the preferred adapter when working with newer Hadoop OutputFormat implementations that use the mapreduce package.
Code Reference
Source Location
- Repository: Apache_Flink
- File: flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopOutputFormat.java
- Lines: 1-53
Signature
@Public
public class HadoopOutputFormat<K, V> extends HadoopOutputFormatBase<K, V, Tuple2<K, V>>
Import
import org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| mapreduceOutputFormat | org.apache.hadoop.mapreduce.OutputFormat<K, V> | Yes | The Hadoop mapreduce OutputFormat instance to wrap |
| job | org.apache.hadoop.mapreduce.Job | Yes | The Hadoop Job instance for configuration |
Outputs
| Name | Type | Description |
|---|---|---|
| record | Tuple2<K, V> | The key-value tuple to be written; f0 is written as the key and f1 as the value through the Hadoop RecordWriter |
Usage Examples
// Create a HadoopOutputFormat wrapping a Hadoop mapreduce TextOutputFormat
import org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.mapreduce.Job;
Job job = Job.getInstance();
TextOutputFormat.setOutputPath(job, new org.apache.hadoop.fs.Path("/path/to/output"));
HadoopOutputFormat<LongWritable, Text> hadoopOF =
new HadoopOutputFormat<>(new TextOutputFormat<LongWritable, Text>(), job);
// Use hadoopOF as a Flink OutputFormat
DataSet<Tuple2<LongWritable, Text>> data = ...;
data.output(hadoopOF);