Workflow:Kubeflow Pipelines XGBoost Training Pipeline
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Training, XGBoost |
| Last Updated | 2026-02-13 14:00 GMT |
Overview
End-to-end machine learning pipeline that loads data, trains XGBoost models, and generates predictions using Kubeflow Pipelines reusable components.
Description
This workflow demonstrates how to build a complete ML training and prediction pipeline using KFP's reusable component system. It loads tabular data from the Chicago Taxi dataset, trains XGBoost regression models in both CSV and Apache Parquet formats, and runs predictions against the trained models. The pipeline showcases cross-format interoperability by verifying that models trained on one format can predict on data in another format.
Key characteristics:
- Uses pre-built, reusable KFP components loaded from URLs
- Demonstrates both CSV and Parquet data format handling
- Trains regression models with configurable hyperparameters
- Validates cross-format prediction compatibility
- Sets memory resource limits on compute-intensive tasks
Usage
Execute this workflow when you need to build an end-to-end ML pipeline that trains an XGBoost model on tabular data and generates predictions. This is the canonical example for learning how KFP orchestrates multi-step ML workflows using reusable components. It is appropriate when you have structured data, need gradient-boosted tree models, and want to leverage KFP's component reuse and artifact management capabilities.
Execution Steps
Step 1: Load Training Data
Fetch structured tabular data from an external data source. In this workflow, the Chicago Taxi dataset is queried with SQL-like filters to select relevant columns (tips, trip metrics, fare information) and limit the result set. The output is a CSV-formatted dataset artifact that downstream steps consume.
Key considerations:
- Specify column selection and row filtering at query time to reduce data transfer
- The data source component is loaded from a versioned URL, ensuring reproducibility
- Output is stored as a KFP artifact for automatic lineage tracking
Step 2: Train Model on CSV Data
Train an XGBoost regression model directly on the CSV-formatted training data. The training component accepts the dataset artifact, a label column index, an objective function, and the number of boosting iterations. The resulting model artifact is persisted for downstream prediction and cross-format validation.
Key considerations:
- The label column is specified by positional index (column 0)
- The objective function (e.g., reg:squarederror) determines the loss function
- Memory limits should be set on training tasks to prevent out-of-memory failures
Step 3: Predict on CSV Data
Run predictions using the trained CSV model against the original CSV training data. This step validates that the model produces coherent outputs and that the prediction component correctly handles model and data artifacts.
Key considerations:
- Prediction reuses the same label column specification as training
- Output predictions are stored as artifacts for evaluation or further processing
Step 4: Convert Data to Parquet Format
Transform the CSV training data into Apache Parquet columnar format. This conversion enables testing the pipeline with a different serialization format, demonstrating KFP's ability to chain format conversion components.
Key considerations:
- Parquet format offers better compression and columnar access patterns
- The conversion component preserves schema information including column names
Step 5: Train Model on Parquet Data
Train a second XGBoost model using the Parquet-formatted dataset. Unlike CSV training, the label column is specified by name rather than index, reflecting Parquet's named column semantics.
Key considerations:
- Label column specified by name (e.g., "tips") instead of index
- Same hyperparameters (objective, iterations) used for fair comparison
Step 6: Cross Format Prediction Validation
Validate model interoperability by running predictions across formats: use the CSV-trained model to predict on Parquet data and vice versa. This confirms that the underlying XGBoost model representation is format-agnostic.
Key considerations:
- Cross-format predictions verify model portability
- Both prediction directions (CSV model on Parquet data, Parquet model on CSV data) are tested