Workflow:Evidentlyai Evidently Data Drift Monitoring
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Data_Monitoring, Drift_Detection |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
End-to-end process for detecting and monitoring data distribution drift in tabular ML datasets using Evidently Reports with drift metrics, storing results in a database, and visualizing them in Grafana.
Description
This workflow outlines the standard procedure for batch monitoring of data drift in production ML systems. It leverages Evidently's drift detection metrics (ValueDrift, DriftedColumnsCount) to compare current data batches against a reference dataset, extracting drift scores and storing them in PostgreSQL for time-series visualization. The process covers data loading, dataset creation with typed column definitions, drift report configuration, batch execution over time windows, metric extraction, database storage, and Grafana dashboard visualization.
Goal: A time-series database of drift scores that can be visualized in Grafana dashboards to detect when production data distributions shift from training data.
Scope: From raw production data and a reference dataset through drift computation to stored metrics and dashboard visualization.
Strategy: Uses statistical drift tests (Jensen-Shannon distance, Wasserstein distance, etc.) applied per-column and aggregated across the dataset, with results inserted into PostgreSQL on a per-batch schedule.
Usage
Execute this workflow when you have a deployed ML model receiving production data in batches and need to detect whether the input data distribution has shifted compared to a reference (training or validation) dataset. This is essential for triggering retraining pipelines, alerting on data quality degradation, or auditing model reliability over time.
Execution Steps
Step 1: Prepare Reference and Production Data
Load the reference dataset (typically training or validation data) and the incoming production data. The reference dataset serves as the baseline distribution that all future data batches will be compared against.
Key considerations:
- Reference data should be representative of the expected production distribution
- Both datasets must share the same column schema
- Handle missing values appropriately before comparison
Step 2: Define Data Schema
Create a DataDefinition that maps each column to its type (numerical, categorical, text) and assigns prediction/target roles. This schema tells Evidently how to select the appropriate statistical test for each column type.
Key considerations:
- Numerical columns use distance-based tests (Wasserstein, KS)
- Categorical columns use divergence-based tests (Jensen-Shannon, chi-squared)
- Prediction and target columns can be tracked separately for model-specific drift
Step 3: Create Evidently Datasets
Wrap both reference and current data as Evidently Dataset objects using the defined schema. This abstraction provides typed column access and ensures consistent processing across all metrics.
Pseudocode:
reference_dataset = Dataset.from_pandas(reference_data, data_definition=schema) current_dataset = Dataset.from_pandas(current_data, data_definition=schema)
Step 4: Configure Drift Report
Assemble a Report with the desired drift metrics. Common choices include ValueDrift for per-column drift scores, DriftedColumnsCount for an aggregate count of shifted columns, and MissingValueCount for data completeness checks.
Key considerations:
- Select drift metrics appropriate to your monitoring goals
- ValueDrift uses the best statistical test per column type by default
- Custom drift methods (PSI, KL divergence, etc.) can be specified via parameters
Step 5: Execute Report Over Data Batches
Run the report for each time window (e.g., daily batches) by slicing the production data into temporal segments and comparing each batch against the fixed reference dataset.
What happens:
- For each batch, the report computes all configured metrics
- Results are returned as a Snapshot object containing metric values
- The snapshot can be exported as a dictionary or JSON for downstream consumption
Step 6: Extract and Store Metrics
Parse the report results dictionary to extract individual metric values (drift scores, column counts, missing value shares) and insert them into a PostgreSQL table with a timestamp for time-series analysis.
Key considerations:
- Create the database table with appropriate column types before inserting
- Include a timestamp column for time-series ordering
- Map metric result paths to the correct database columns
Step 7: Visualize in Grafana
Configure Grafana to connect to the PostgreSQL database and create dashboard panels that plot drift scores over time, enabling visual monitoring of data distribution changes.
Key considerations:
- Set up PostgreSQL as a Grafana data source
- Create line plots for drift score trends per feature
- Add threshold alerts for drift scores exceeding acceptable limits