Principle:Apache Airflow Component Deployment Scaling
| Knowledge Sources | |
|---|---|
| Domains | Kubernetes, Scaling |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
A Kubernetes deployment pattern for running and scaling individual Airflow components as separate workloads with configurable replicas and resources.
Description
Component Deployment and Scaling defines how each Airflow component (scheduler, webserver, API server, workers, triggerer, dag-processor, flower) is deployed as a separate Kubernetes workload. The Helm chart generates 103 template files producing Deployments, StatefulSets, CronJobs, Services, HPAs, and PodDisruptionBudgets. Each component can be independently scaled, resourced, and configured.
Usage
Configure component replicas and resources in values.yaml. Enable HPA (Horizontal Pod Autoscaler) for dynamic scaling of workers based on load. Use PodDisruptionBudgets for safe rollouts.
Theoretical Basis
Component Architecture:
- Scheduler: 1+ replicas (HA supported), Deployment or StatefulSet
- Webserver/API Server: 1+ replicas behind Service/Ingress
- Workers: 1+ replicas, autoscalable via HPA (Celery only)
- Triggerer: 1+ replicas for async trigger handling
- Dag-processor: 1 replica for DAG file parsing
- Redis: StatefulSet (Celery broker)