Implementation:Apache Flink FromElementsGeneratorFunction
| Knowledge Sources | |
|---|---|
| Domains | Connectors, Source_Framework |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A stream generator function that returns a sequence of elements from a pre-defined collection, serialized using Flink's type information framework.
Description
FromElementsGeneratorFunction is an internal implementation of the GeneratorFunction interface that produces output elements by sequentially deserializing from a pre-serialized byte array of elements. During construction, the provided collection of elements is serialized into a compact byte array using Flink's TypeSerializer. At runtime, the map() method deserializes elements on demand, supporting forward-seeking to handle failure recovery scenarios where the index may jump ahead past previously emitted elements.
The class also implements OutputTypeConfigurable, which allows the output type serializer to be overridden after construction. This supports legacy usage patterns from StreamExecutionEnvironment#fromElements() where type information may not be available at construction time but is supplied later via the returns() method.
Key design decisions:
- Elements are serialized eagerly during construction to avoid Java serialization issues when the function object is shipped to the cluster.
- The function maintains an internal counter (numElementsEmitted) to track position and supports seeking forward during recovery.
- Null elements and type mismatches are validated at construction time via checkIterable().
Usage
Use FromElementsGeneratorFunction when you need to create a DataGeneratorSource that emits a fixed, known collection of elements in sequential order. This is the backing implementation for StreamExecutionEnvironment.fromElements() and fromCollection() operations. It is marked @Internal and is not intended for direct use by application developers.
Code Reference
Source Location
- Repository: Apache_Flink
- File:
flink-connectors/flink-connector-datagen/src/main/java/org/apache/flink/connector/datagen/functions/FromElementsGeneratorFunction.java - Lines: 1-201
Signature
@Internal
public class FromElementsGeneratorFunction<OUT>
implements GeneratorFunction<Long, OUT>, OutputTypeConfigurable<OUT>
Import
import org.apache.flink.connector.datagen.functions.FromElementsGeneratorFunction;
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| typeInfo | TypeInformation<OUT> | Yes | Type information used to create the serializer for elements |
| elements | Iterable<OUT> or OUT[] (varargs) | Yes | The collection of elements to emit; must be non-null and type-compatible |
| config | ExecutionConfig | No | Execution configuration for serializer creation (defaults to new ExecutionConfig) |
| nextIndex (map input) | Long | Yes | The sequential index of the next element to produce |
Outputs
| Name | Type | Description |
|---|---|---|
| map result | OUT | The deserialized element at the requested index position |
Usage Examples
// Create a generator function from a list of strings
TypeInformation<String> typeInfo = Types.STRING;
FromElementsGeneratorFunction<String> generatorFunction =
new FromElementsGeneratorFunction<>(typeInfo, "alpha", "beta", "gamma");
// Use with DataGeneratorSource to produce exactly 3 elements
DataGeneratorSource<String> source =
new DataGeneratorSource<>(generatorFunction, 3, Types.STRING);
DataStreamSource<String> stream =
env.fromSource(source, WatermarkStrategy.noWatermarks(), "Elements Source");