Workflow:SeldonIO Seldon core Production Monitoring Pipeline
| Knowledge Sources | |
|---|---|
| Domains | MLOps, Model_Monitoring, Data_Science, Kubernetes |
| Last Updated | 2026-02-13 14:00 GMT |
Overview
End-to-end process for deploying a production ML pipeline with integrated drift detection, outlier detection, and preprocessing in Seldon Core 2.
Description
This workflow covers building a production-grade inference pipeline that goes beyond simple prediction by incorporating data quality monitors. The pipeline chains together a preprocessor for feature engineering, the core classifier model, an outlier detector that flags anomalous inputs, and a drift detector that identifies distribution shifts in incoming data over time. All components communicate via Kafka-based data flow, with drift detection optionally running in batch mode to aggregate statistical tests over multiple requests. The pipeline exposes both predictions and monitoring flags in a single response.
Usage
Execute this workflow when deploying a model to production where data quality assurance is critical. Typical triggers include regulatory requirements for model monitoring, high-stakes predictions (finance, healthcare), or any deployment where input data distributions may shift over time. This workflow is appropriate when you have pre-trained drift and outlier detection models (commonly built with the Alibi Detect library) alongside your production classifier.
Execution Steps
Step 1: Train Monitoring Models
Train the drift detector and outlier detector models using the same training data distribution as the production classifier. The drift detector learns the reference distribution to compare against incoming data, while the outlier detector learns the boundary of normal inputs. Both are typically built using the Alibi Detect library and saved as serialized model artifacts alongside their MLServer model-settings.json configurations.
Key considerations:
- Drift detectors use statistical tests (Chi-Square for categorical, Kolmogorov-Smirnov for continuous features)
- Outlier detectors commonly use Variational Autoencoders (VAE) or isolation forests
- Both detectors must be trained on representative samples of the expected production data distribution
- Save detector artifacts in the same format expected by MLServer's alibi-detect runtime
Step 2: Deploy All Pipeline Components
Deploy each component model individually: the preprocessor, the core classifier, the outlier detector, and the drift detector. Each component is a separate Seldon Model resource with its own storage URI, framework requirements, and memory allocation. Wait for all models to reach the Available state.
Key considerations:
- The preprocessor typically requires the python or sklearn runtime
- The classifier uses the appropriate framework runtime (sklearn, xgboost, tensorflow, etc.)
- Drift and outlier detectors require the alibi-detect runtime capability
- All components must be deployed before the pipeline can become ready
Step 3: Define Monitoring Pipeline
Create a Pipeline resource that connects the components in the correct topology. The preprocessor feeds into both the classifier and the outlier detector. The drift detector can run asynchronously in batch mode on the preprocessed data. The output section selects which results to include in the response (typically the classifier prediction and outlier/drift flags).
Key considerations:
- Use the batch configuration on the drift detector step to aggregate multiple requests before running the statistical test
- Configure output steps to include specific tensor outputs (e.g., income-outlier.outputs.is_outlier)
- The preprocessor step may need tensor mapping if the classifier expects different input names
- Drift detection can run asynchronously without blocking the prediction response
Step 4: Deploy and Validate Pipeline
Apply the Pipeline resource and wait for it to reach PipelineReady status. Validate the pipeline by sending test requests with known data and verifying that predictions, outlier flags, and drift indicators are all present in the response.
Key considerations:
- Verify that non-outlier inputs return is_outlier=0 and outlier inputs return is_outlier=1
- Drift detection may require multiple requests before producing results (due to batch size)
- Use pipeline inspect to observe data flowing through individual steps for debugging
- Check that all tensor mappings are correctly routing data between steps
Step 5: Monitor Production Traffic
Once deployed, the pipeline continuously monitors production traffic. Outlier detection runs on every request, providing immediate flags for anomalous inputs. Drift detection accumulates batches and periodically evaluates whether the input distribution has shifted from the training reference.
Key considerations:
- Outlier flags should be logged and can trigger alerts via external monitoring systems
- Drift detection results indicate statistical significance of distribution changes
- Prometheus metrics track per-step latency and throughput for operational monitoring
- Consider setting up alerting on drift/outlier detection rates exceeding thresholds
Step 6: Iterate on Pipeline Versions
As models are retrained or monitoring thresholds are adjusted, update individual components without redeploying the entire pipeline. Pipeline versions can be updated in place by modifying the Pipeline resource, enabling progressive addition of monitoring capabilities (e.g., starting with just a classifier, then adding outlier detection, then drift detection).
Key considerations:
- Individual model updates do not require pipeline redeployment
- Pipeline topology changes (adding/removing steps) require a pipeline resource update
- Version the pipeline manifest to track monitoring capability evolution
- Retrain detectors periodically as the production data distribution naturally evolves