Principle:Dagster io Dagster Resource Management
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Databases |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Mechanism for managing external service connections and shared configuration across assets through dependency-injected resource objects.
Description
Resources in Dagster are configurable objects that provide connections to external systems such as databases, APIs, and cloud services. They are defined once in a central Definitions object and injected into assets and ops via parameter names matching resource keys. This decouples business logic from infrastructure configuration, enabling environment-specific configuration (development, staging, production) without code changes.
A resource encapsulates:
- Configuration: Typed parameters (connection strings, credentials, timeouts) validated at launch time.
- Lifecycle Management: Setup and teardown logic (e.g., opening and closing database connections).
- Shared State: A single resource instance can be shared across multiple assets within a run, ensuring consistent connections.
Resources are registered in the Definitions object with string keys. When an asset function declares a parameter whose name matches a resource key, Dagster automatically injects the configured resource instance.
Usage
Use resources when assets need to interact with external systems. Common examples include database connections (DuckDB, PostgreSQL, Snowflake), API clients (OpenAI, Slack, AWS), file storage systems (S3, GCS), and any other shared service. Resources centralize connection configuration and enable environment-based overrides through resource key remapping.
Theoretical Basis
Resources implement the Dependency Injection pattern. Instead of assets creating their own connections to external systems, the framework injects pre-configured service objects at runtime. This follows the Inversion of Control principle, where the framework (not user code) controls object lifecycle and configuration.
The key theoretical advantages of this approach:
- Loose Coupling: Asset code depends on abstract resource interfaces, not concrete implementations. A DuckDB resource in development can be swapped for a Snowflake resource in production.
- Testability: Resources can be replaced with mocks or stubs during testing without modifying asset code.
- Configuration Centralization: All external system configuration lives in one place (the Definitions object), making environment management straightforward.
# Pseudocode illustrating the DI pattern
# 1. Define the resource (configuration + lifecycle)
resource = DatabaseResource(connection_string="...")
# 2. Register it with a key
definitions = Definitions(resources={"db": resource})
# 3. Asset receives it via parameter injection
def my_asset(db: DatabaseResource):
db.query("SELECT ...")