Principle:Apache Flink Fatal Exception Classification
| Knowledge Sources | |
|---|---|
| Domains | Stream_Processing, Error_Handling |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
An exception classification mechanism that distinguishes fatal errors (requiring job failure) from retryable errors (allowing re-buffering) in asynchronous sink operations.
Description
Fatal Exception Classification solves the problem of determining the correct response to async I/O failures. Not all errors are equal:
- Fatal errors: Authentication failures, permission errors, schema mismatches — these will never succeed on retry and should immediately fail the job
- Retryable errors: Throttling, temporary unavailability, transient network issues — these should trigger re-buffering of failed entries
The classifier uses a chain-of-responsibility pattern where multiple classifiers can be composed. Each classifier has a Predicate<Throwable> to match and a Function<Throwable, Exception> to map the error. The framework inspects the root cause to handle wrapped exceptions.
Usage
Implement a FatalExceptionClassifier when building a custom async sink to define which destination-specific exceptions are fatal. Unclassified exceptions default to being retryable.
Theoretical Basis
// Abstract classification algorithm
function classify(throwable):
for each classifier in chain:
rootCause = findRootCause(throwable)
if classifier.matches(rootCause):
return FATAL(classifier.map(rootCause))
return RETRYABLE // default: retry