Principle:Dagster io Dagster LLM Fine Tuning Orchestration
| Property | Value |
|---|---|
| Type | Principle |
| Category | AI, NLP, Fine_Tuning |
| Repository | Dagster_io_Dagster |
| Related Implementation | Implementation:Dagster_io_Dagster_OpenAI_Fine_Tuning_Pattern |
Overview
Pattern for orchestrating LLM fine-tuning workflows including data preparation, file upload, job management, and model validation as Dagster assets.
Description
LLM fine-tuning orchestration models the entire fine-tuning workflow as a DAG of assets:
- Data ingestion -- Raw data collection and loading
- Feature engineering -- Transforming raw data into training features
- Training file creation -- Converting features to the required JSONL format
- File upload -- Uploading training/validation files to the LLM provider API
- Fine-tuning job -- Creating and monitoring the asynchronous fine-tuning job
- Model validation -- Comparing fine-tuned model accuracy against the base model
Each stage is an asset with clear inputs/outputs. The pattern handles OpenAI's asynchronous fine-tuning API (polling for completion), JSONL format requirements, and automated model comparison (fine-tuned vs. base model accuracy). Asset checks provide automated quality validation after the fine-tuning completes.
Usage
Use when fine-tuning an LLM (OpenAI, etc.) as part of a data pipeline. The pattern handles:
- Data preparation and format validation (JSONL conversion)
- Asynchronous job management with polling
- Automated quality validation via asset checks
- A/B comparison between fine-tuned and base models
This is applicable whenever you need to customize a foundation model on domain-specific data and want reproducible, observable fine-tuning pipelines.
Theoretical Basis
LLM fine-tuning orchestration applies the ETL pattern to ML workflows:
- Extract -- Ingest raw data (e.g., book reviews, domain text)
- Transform -- Feature engineering, format conversion to JSONL training format
- Load -- Upload to training API (OpenAI files endpoint)
The fine-tuning job itself is an external computation orchestrated through polling (check status -> sleep -> check again). This follows the async command pattern: the Dagster asset submits a job and polls for completion rather than performing the computation directly.
Model validation implements A/B testing by comparing fine-tuned model accuracy against the base model on a holdout set. The asset check returns a pass/fail result with severity metadata, enabling automated quality gates in the pipeline. If the fine-tuned model does not outperform the base model, the check emits a warning, preventing silent regression.