Workflow:Kubeflow Pipelines Iterative Model Training
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Training, Model_Evaluation |
| Last Updated | 2026-02-13 14:00 GMT |
Overview
Continuous training pipeline that uses a recursive train-evaluate-check loop to iteratively improve a model until evaluation metrics meet a quality threshold.
Description
This workflow demonstrates an iterative model improvement pattern using KFP's recursive graph component capability. The pipeline trains an initial XGBoost model, then enters a recursive loop that retrains the model, evaluates predictions against ground truth, calculates regression metrics, and checks whether the error is below an acceptable threshold. If the model does not yet meet quality requirements, training continues with the existing model as a starting point. This pattern is common in production ML systems where model quality gates must be met before deployment.
Key characteristics:
- Uses @dsl.graph_component for recursive sub-pipeline definition
- Implements train-predict-evaluate-check loop with conditional termination
- Warm-starts each training iteration from the previous model's weights
- Uses regression metrics (mean squared error) as the quality gate
- Automatically terminates when the quality threshold is satisfied
Usage
Execute this workflow when you need a model to meet specific quality thresholds before the pipeline completes. This is appropriate for automated retraining scenarios where model quality must be guaranteed, and additional training iterations should be applied until metrics converge. It applies to regression tasks where mean squared error (or similar metrics) can serve as the stopping criterion.
Execution Steps
Step 1: Prepare Training Data
Load and prepare the training dataset. The raw data is fetched from an external source (e.g., Chicago Taxi dataset), selecting relevant feature columns and the target variable. Additionally, extract the ground truth labels into a separate artifact for use in the evaluation step.
Key considerations:
- Select features and target columns at query time
- Use pandas transform components to extract ground truth labels separately
- Drop headers from the ground truth CSV for metric calculation compatibility
- The training data artifact is reused across all recursive iterations
Step 2: Train Initial Model
Train the first XGBoost model from scratch on the prepared training data. This initial training uses a higher number of boosting iterations (e.g., 100) to establish a baseline model. The resulting model artifact is passed into the recursive improvement loop.
Key considerations:
- Initial training starts without a pre-existing model
- Higher iteration count for the first training pass establishes a strong baseline
- The model artifact output becomes the starting point for recursive refinement
Step 3: Enter Recursive Training Loop
Invoke the recursive sub-pipeline that implements the train-evaluate-check cycle. The sub-pipeline is defined as a @dsl.graph_component and receives the current model, training data, and ground truth labels as inputs.
Key considerations:
- The graph_component decorator enables recursive self-invocation
- Each recursive call creates a new set of training, prediction, and evaluation tasks
- The loop terminates when the quality condition is no longer met
Step 4: Retrain from Existing Model
Within the recursive loop, continue training the model from its current state. The starting_model parameter enables warm-starting, where the existing model weights are used as initialization for additional boosting iterations (e.g., 50 per cycle).
Key considerations:
- Warm-starting preserves knowledge from previous training iterations
- Fewer boosting iterations per cycle (e.g., 50 vs 100 for initial) for incremental improvement
- The output model replaces the input model for the next evaluation
Step 5: Evaluate and Check Metrics
Generate predictions using the retrained model, compare them against ground truth labels, and calculate regression metrics. The mean squared error is compared against a threshold (e.g., 0.01) to determine whether the model meets quality requirements.
Key considerations:
- Predictions are generated on the full training data for evaluation
- Regression metrics include mean squared error, mean absolute error, and others
- The threshold condition (MSE > 0.01) gates whether another training cycle is needed
- If the condition is true (error still too high), the recursive call triggers another cycle
Step 6: Terminate or Continue
Based on the metric check, either terminate the loop (metrics pass) or recursively invoke the sub-pipeline again with the updated model. When the quality threshold is met, the recursion unwinds and the pipeline completes with the final trained model.
Key considerations:
- The dsl.Condition guards the recursive call, providing automatic termination
- The final model meets the quality guarantee encoded in the threshold
- Pipeline execution history shows the full sequence of training iterations