Workflow:Evidentlyai Evidently Text Data Quality Evaluation
| Knowledge Sources | |
|---|---|
| Domains | NLP, Data_Quality, Text_Analysis |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
End-to-end process for evaluating text data quality using Evidently's descriptor framework to compute row-level text properties (length, sentiment, OOV percentage, regex patterns, trigger words) and summarize them in a report.
Description
This workflow covers the standard procedure for assessing the quality of text data in datasets used for NLP or LLM applications. It uses Evidently's descriptor system to compute row-level text features such as character length, sentiment polarity, out-of-vocabulary word percentage, URL detection via regex, and trigger word presence. These computed features are then summarized using the TextEvals preset or individual statistics metrics (MeanValue, InRangeValueCount, CategoryCount) to produce aggregate quality reports.
Goal: A comprehensive text quality report showing distributions and statistics of text properties across the dataset, enabling identification of data quality issues before model training or during production monitoring.
Scope: From raw text data through descriptor-based feature extraction to aggregated quality reports with optional drift detection.
Strategy: Uses row-level descriptors to compute text features, then applies statistical metrics to summarize and compare distributions, optionally with reference data for drift detection.
Usage
Execute this workflow when you have a text dataset (reviews, user inputs, LLM prompts/responses) and need to assess its quality characteristics before training or during production. This is useful for detecting anomalies like unusual text lengths, sentiment shifts, increases in out-of-vocabulary words, or the appearance of unwanted patterns (URLs, competitor mentions).
Execution Steps
Step 1: Load Text Data
Load the text dataset containing the columns to evaluate. Prepare both current and optionally reference datasets for comparison.
Key considerations:
- Identify which columns contain text data to evaluate
- Reference data enables drift detection on text properties
- Handle missing text values (null or empty strings) before processing
Step 2: Define Data Schema with Text Columns
Create a DataDefinition that explicitly marks text columns. This ensures Evidently applies text-appropriate processing and enables text-specific descriptors.
Key considerations:
- Mark text columns explicitly in the DataDefinition
- Also define numerical and categorical columns for any non-text features
- If the dataset includes model predictions, add classification or regression configuration
Step 3: Configure Text Descriptors
Select and configure the descriptors that define which text properties to compute. Each descriptor targets a specific text column and produces a new column with the computed property.
Key considerations:
- TextLength computes character count per row
- Sentiment uses NLTK VADER for polarity scoring (-1 to 1)
- OOVWordsPercentage detects non-standard vocabulary
- RegExp matches custom patterns (e.g., URLs, email addresses)
- TriggerWordsPresent checks for specific word lists (e.g., competitor names)
- NonLetterCharacterPercentage measures text cleanliness
- Assign readable aliases for clear output column names
Step 4: Create Evaluated Datasets
Wrap the dataframes as Evidently Datasets with the configured descriptors. The Dataset constructor computes all descriptor values, producing a dataframe with additional columns for each text property.
Pseudocode:
dataset = Dataset.from_pandas(
text_data,
data_definition=schema,
descriptors=[text_length, sentiment, oov, regex_urls, trigger_words]
)
Step 5: Build Quality Report
Configure a Report using the TextEvals preset for automatic summarization of all descriptor columns, or use individual metrics (MeanValue, InRangeValueCount, CategoryCount, ValueDrift) on descriptor-generated columns for targeted analysis.
Key considerations:
- TextEvals preset auto-summarizes all descriptor columns in the dataset
- Individual metrics allow specific thresholds and comparisons
- ValueDrift on descriptor columns detects shifts in text properties between current and reference
- include_tests=True enables automatic quality thresholds
Step 6: Run and Analyze Report
Execute the report with current data and optional reference data. Review the results to identify text quality issues, distribution shifts, or violations of expected text properties.
What happens:
- All descriptor column distributions are computed and compared
- Drift detection identifies which text properties have shifted
- Test conditions flag specific quality violations
- Results are viewable as interactive HTML or extractable as JSON/dictionary