Workflow:Scikit learn Scikit learn Data Preprocessing Pipeline
| Knowledge Sources | |
|---|---|
| Domains | Machine_Learning, Data_Engineering, Feature_Engineering |
| Last Updated | 2026-02-08 15:00 GMT |
Overview
End-to-end process for building a reusable data preprocessing pipeline that handles heterogeneous feature types, applies column-specific transformations, and chains preprocessing with a final estimator.
Description
This workflow demonstrates how to construct scikit-learn Pipelines and ColumnTransformers to create reproducible, leak-free data processing workflows. It covers handling mixed data types (numerical and categorical), applying appropriate transformations to each feature group, imputing missing values, and composing all steps into a single Pipeline object that can be fitted and used for prediction as a unified estimator. This pattern is essential for production-ready machine learning systems.
Usage
Execute this workflow when you have heterogeneous tabular data with mixed feature types (numerical, categorical, text) that require different preprocessing transformations before model training. This is the standard approach for real-world datasets where raw data needs cleaning, imputation, encoding, and scaling.
Execution Steps
Step 1: Data Inspection
Load the dataset and examine feature types, missing value patterns, and value distributions. Identify which columns are numerical, which are categorical, and which may require special handling. This informs the design of the preprocessing pipeline.
Key considerations:
- Use pandas DataFrame column dtypes to classify features automatically
- Identify columns with missing values that need imputation
- Note cardinality of categorical features (low vs. high cardinality affects encoding choice)
Step 2: Define Column Groups
Partition features into groups that will receive the same preprocessing treatment. Use make_column_selector or explicit column name lists to define numerical columns and categorical columns. This grouping drives the ColumnTransformer configuration.
Key considerations:
- make_column_selector can auto-detect columns by dtype pattern
- Explicit column lists are more maintainable for production pipelines
- Consider creating additional groups for text, datetime, or ordinal features
Step 3: Build Feature Transformers
Create sub-pipelines for each feature group. A typical numerical pipeline includes imputation followed by scaling. A typical categorical pipeline includes imputation followed by one-hot or ordinal encoding. Each sub-pipeline is itself a Pipeline object.
Key considerations:
- Numerical: SimpleImputer (median/mean) then StandardScaler or MinMaxScaler
- Categorical: SimpleImputer (most_frequent/constant) then OneHotEncoder or OrdinalEncoder
- Set handle_unknown appropriately to handle unseen categories at prediction time
Step 4: Compose with ColumnTransformer
Combine the feature-group sub-pipelines into a ColumnTransformer that routes each column group to its corresponding transformer. The ColumnTransformer applies transformations in parallel and concatenates the results into a single feature matrix.
Key considerations:
- Use the remainder parameter to handle columns not explicitly assigned (drop or passthrough)
- Sparse output is used by default for memory efficiency when possible
- The transformer names are used for accessing feature names after fitting
Step 5: Chain with Final Estimator
Wrap the ColumnTransformer and a final estimator (classifier or regressor) into an outer Pipeline. This ensures that all preprocessing and prediction happen as a single atomic operation, preventing data leakage during cross-validation and hyperparameter search.
Key considerations:
- The pipeline can be passed directly to cross_validate or GridSearchCV
- Pipeline parameter access uses double-underscore syntax (e.g., classifier__C)
- set_output can configure pandas DataFrame output from intermediate steps
Step 6: Fit and Predict
Train the full pipeline on training data and generate predictions on test data. The pipeline orchestrates the sequential execution of all preprocessing steps followed by the final estimator, handling all data transformations automatically.
Key considerations:
- Only call fit on training data; transform is applied to test data automatically via predict
- The fitted pipeline can be serialized with joblib for deployment
- Feature names are propagated through the pipeline when using DataFrame input