Heuristic:Dagster io Dagster Batch Size Tuning
| Knowledge Sources | |
|---|---|
| Domains | Performance, Tuning |
| Last Updated | 2026-02-10 12:00 GMT |
Overview
Environment variable-based tuning of batch sizes and concurrency limits for backfills, event processing, and multiprocess execution.
Description
Dagster exposes multiple environment variables for tuning batch processing sizes and concurrency limits. These are critical for production performance tuning because the defaults are conservative and may not be optimal for large-scale deployments. The batch sizes control how many operations are processed in a single database round-trip or execution cycle, directly impacting throughput and memory usage.
Usage
Use this heuristic when experiencing slow backfills, event processing bottlenecks, or executor concurrency issues in production Dagster deployments. These tuning knobs are particularly important for deployments with thousands of assets, high-frequency sensors, or large partition sets.
The Insight (Rule of Thumb)
- Backfill Run Chunk Size:
- Variable:
DAGSTER_ASSET_BACKFILL_RUN_CHUNK_SIZE - Default: 25
- Action: Increase for faster backfills with many partitions. Decrease if database connections are limited.
- Trade-off: Higher values = faster backfills but more concurrent database load.
- Variable:
- Backfill Materialization Chunk Size:
- Variable:
DAGSTER_ASSET_BACKFILL_MATERIALIZATION_CHUNK_SIZE - Default: 1000
- Action: Increase if backfill status checking is slow. Decrease if memory is constrained.
- Trade-off: Higher values = fewer database queries but more memory per query.
- Variable:
- Multiprocess Executor Concurrency:
- Variable:
DAGSTER_MULTIPROCESS_EXECUTOR_MAX_CONCURRENT - Default:
multiprocessing.cpu_count() - Action: Set lower than CPU count to leave headroom for system processes. Set higher for I/O-bound workloads.
- Trade-off: Higher values = more parallelism but more memory and CPU contention.
- Variable:
- Event Batch Size:
- Variable:
DAGSTER_EVENT_BATCH_SIZE - Default: 0 (disabled, events stored individually)
- Action: Set to a positive integer (e.g., 100) to batch event storage. Reduces database round-trips.
- Trade-off: Events may be delayed in storage until the batch is full.
- Variable:
- Executor Event Pop Limit:
- Variable:
DAGSTER_EXECUTOR_POP_EVENTS_LIMIT - Default: 1000
- Action: Increase for very high-throughput pipelines. Decrease for lower-latency event processing.
- Variable:
- Backfill Cancel Batch Size:
- Variable:
DAGSTER_BACKFILL_CANCEL_RUNS_BATCH_SIZE - Default: 500
- Action: Increase for faster cancellation of large backfills. Decrease if database load is a concern.
- Variable:
Reasoning
The default batch sizes in Dagster are deliberately conservative to work well across a wide range of deployments, from single-machine SQLite setups to large PostgreSQL clusters. However, production deployments with significant scale often benefit from tuning these values:
- Database I/O: Larger batch sizes reduce the number of database round-trips, which is critical when database latency is the bottleneck (common with remote PostgreSQL).
- Memory trade-off: Each batch is held in memory until processed. Very large batch sizes can cause memory pressure on the daemon process.
- Concurrency vs. contention: The multiprocess executor defaults to CPU count, but I/O-bound workloads (API calls, file processing) can safely use higher concurrency, while CPU-bound workloads may need lower values.
Code Evidence
Backfill batch sizes from asset_backfill.py:74-80:
def get_asset_backfill_run_chunk_size():
return int(os.getenv("DAGSTER_ASSET_BACKFILL_RUN_CHUNK_SIZE", "25"))
MATERIALIZATION_CHUNK_SIZE = int(
os.getenv("DAGSTER_ASSET_BACKFILL_MATERIALIZATION_CHUNK_SIZE", "1000")
)
Multiprocess concurrency from multiprocess.py:130-132:
max_concurrent = int(os.getenv(
"DAGSTER_MULTIPROCESS_EXECUTOR_MAX_CONCURRENT",
str(multiprocessing.cpu_count())
))