Principle:Dagster io Dagster Run Configuration
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Configuration |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Mechanism for parameterizing asset and op computations at runtime through type-safe, validated configuration objects.
Description
Run configuration allows users to provide runtime parameters when materializing assets. Using Pydantic-based Config classes, Dagster validates configuration at launch time and makes parameters available in the Dagster UI as auto-generated form fields. This separates runtime parameters (batch sizes, thresholds, model names, file paths) from asset definitions, enabling the same asset code to serve different use cases without modification.
The configuration system provides:
- Type Safety: Configuration fields are typed using standard Python type annotations and validated by Pydantic at launch time.
- UI Integration: Config classes automatically generate form fields in the Dagster UI, allowing users to specify parameters without writing code.
- Default Values: Fields can have default values, making configuration optional for common cases.
- Nested Configuration: Config classes can contain nested Pydantic models for complex configuration hierarchies.
Usage
Use run configuration when asset computations need runtime parameterization. Common use cases include model hyperparameters in ML pipelines, file paths or URLs for data ingestion, threshold values for quality checks, sample sizes for development/testing, and any parameter that should be adjustable at materialization time via the UI.
Theoretical Basis
Configuration as data follows the separation of concerns principle. By externalizing parameters into typed configuration schemas, the system enables runtime flexibility while maintaining type safety. The Pydantic-based approach provides validation, documentation, and UI generation from a single source of truth -- the Config class definition.
This approach aligns with the Schema-First Design methodology:
# Pseudocode illustrating the configuration pattern
class MyConfig:
threshold: float = 0.5 # default value
batch_size: int = 1000 # default value
model_name: str # required, no default
# The framework:
# 1. Validates user-provided config against the schema
# 2. Generates UI form fields from the schema
# 3. Injects the validated config object into the asset function
def my_asset(config: MyConfig):
# config.threshold, config.batch_size, config.model_name
# are all type-checked and validated
...
The theoretical benefits include:
- Fail-Fast Validation: Invalid configuration is rejected before any computation begins, preventing wasted execution time.
- Self-Documenting: The Config class serves as documentation for available parameters, their types, and defaults.
- Reproducibility: Configuration is serialized and stored with each run, enabling exact reproduction of past computations.