Principle:Dagster io Dagster Custom Resource Pattern

Field	Value
Principle Name	Custom Resource Pattern
Category	Resource Management
Domains	Data_Engineering, API_Integration
Repository	dagster-io/dagster

Overview

Pattern for building project-specific resource classes that wrap external service clients with Dagster-compatible configuration and lifecycle management.

Description

Custom resources extend Dagster's ConfigurableResource base class to create project-specific wrappers for external services not covered by official integrations. Resources declare configuration fields (API keys, endpoints, credentials) as typed Pydantic attributes, support EnvVar for secret management, and provide methods for obtaining authenticated client instances. This pattern is used for services like Bluesky (ATProtoResource), Pinecone (PineconeResource), GitHub (GithubResource), and DSPy (DSPyResource).

The pattern provides several key capabilities:

• Typed configuration -- Pydantic fields provide validation, type checking, and documentation for all configuration parameters.
• Secret management -- EnvVar references allow secrets to be resolved from environment variables at runtime, never hardcoded.
• UI visibility -- Configuration fields appear in the Dagster UI, making it easy to inspect resource settings without reading code.
• Dependency injection -- Resources are injected into assets by matching parameter names to resource keys in the Definitions object.
• Client lifecycle -- Resources manage client creation, session caching, and cleanup, ensuring efficient use of connections and authentication tokens.

Usage

Use when you need to integrate with an external service that doesn't have an official dagster-* integration library, or when you need a custom wrapper with project-specific logic (e.g., model persistence, index management). Common scenarios:

• Custom API clients -- Wrapping internal or niche APIs that lack official Dagster integrations
• Enhanced wrappers -- Adding project-specific methods (model saving, index creation) on top of base SDK functionality
• Multi-service composites -- Combining multiple related API clients into a single logical resource
• Configuration-driven behavior -- Using resource configuration to control which model, index, or endpoint is used

Theoretical Basis

Custom resources implement the adapter pattern, wrapping external SDKs in a Dagster-compatible interface. The ConfigurableResource base provides:

• Pydantic validation for configuration
• EnvVar support for secrets
• UI-visible configuration fields
• Dependency injection into assets

This follows the ports-and-adapters (hexagonal) architecture pattern where the resource is the adapter between the external service (port) and the application (Dagster assets). The key theoretical properties are:

• Adapter pattern -- The resource adapts the external SDK's interface to Dagster's resource protocol, allowing the SDK to be used transparently within the framework.
• Hexagonal architecture -- Business logic (assets) depends on abstract resource interfaces, not concrete external SDK classes. The resource provides the concrete implementation.
• Single responsibility -- Each resource encapsulates one external service concern, keeping asset code focused on business logic.

Related Pages

• Implementation:Dagster_io_Dagster_ConfigurableResource_Base