Principle:Apache Airflow Operator Sensor Development
| Knowledge Sources | |
|---|---|
| Domains | Workflow_Orchestration, Plugin_System |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
A framework for creating custom task types as operators (action-oriented) and sensors (condition-waiting) that integrate with external systems.
Description
Operator and Sensor Development defines the patterns for creating reusable task components. Operators perform actions (execute queries, transfer data, trigger jobs) by implementing the execute() method. Sensors wait for conditions by implementing the poke() method and can operate in poke mode (blocking) or reschedule mode (yielding worker slots). Both inherit from BaseOperator with extensive configuration for retries, pools, trigger rules, and callbacks.
Usage
Create custom operators when you need a reusable task type for a specific action. Create sensors when tasks need to wait for external conditions. Choose reschedule mode for sensors with long wait times to conserve worker resources.
Theoretical Basis
Operator Lifecycle:
- pre_execute: Optional setup hook
- execute(context): Main task logic — must be implemented
- post_execute: Optional cleanup hook
- on_success_callback / on_failure_callback: State change hooks
Sensor Modes:
- poke: Continuously checks condition in a loop (holds worker slot)
- reschedule: Checks once, then frees worker slot until next check
Template Rendering:
- Operators define template_fields — fields that support Jinja2 templating
- Templates are rendered at execution time with the task context