Implementation:Evidentlyai Evidently Core Tests

Knowledge Sources	Evidentlyai_Evidently
Domains	Testing, Metrics, Data Monitoring
Last Updated	2026-02-14 12:00 GMT

Overview

Core test infrastructure providing threshold types, a unified GenericTest interface, and a Reference model for comparing metric/descriptor values against expected thresholds.

Description

This module defines the foundational types and classes for Evidently's test system:

Reference is a Pydantic BaseModel used in threshold comparisons. It can hold a relative change (as a decimal percentage) and/or an absolute difference threshold. It is hashable and used to compare metric values against reference data.

ThresholdType and ThresholdValue are type aliases defining the valid types for test thresholds: float, int, ApproxValue, and Reference.

threshold_typecheck_guard(value) is a validation function that raises ValueError if a value matches a valid threshold type (note: the logic appears inverted -- it raises when the value is a threshold type).

GenericTest is the central test class that provides a unified interface for both metric-level and descriptor-level tests. Each GenericTest instance contains:

test_name -- Identifies the test type (e.g., "gt", "lt", "between")
metric -- Optional MetricTest implementation
descriptor -- Optional DescriptorTest implementation

GenericTest provides accessor methods for_metric() and for_descriptor() plus a set of bind_* methods that create BoundTest instances for different metric result shapes:

bind_single -- For single-value metrics
bind_count -- For count/share metrics
bind_by_label -- For per-label metrics
bind_by_label_count -- For per-label count/share metrics
bind_mean_std -- For mean/standard-deviation metrics
bind_dataframe -- For dataframe column metrics with optional label filters

Usage

Use GenericTest when defining tests that should work with both metric and descriptor results. Use Reference to define reference-based thresholds for comparison tests. The bind_* methods are used internally by the framework to connect tests to specific metric result types during report execution.

Code Reference

Source Location

Repository: Evidentlyai_Evidently
File: src/evidently/core/tests.py

Signature

class Reference(BaseModel):
    relative: Optional[float] = Field(default=None)
    absolute: Optional[float] = Field(default=None)

ThresholdType = Union[float, int, ApproxValue, Reference]
ThresholdValue = Union[float, int, ApproxValue]

def threshold_typecheck_guard(value: Any):

class GenericTest(BaseModel):
    test_name: str
    metric: Optional["MetricTest"]
    descriptor: Optional["DescriptorTest"]

    def for_metric(self) -> "MetricTest":
    def for_descriptor(self) -> "DescriptorTest":
    def bind_single(self, fingerprint: Fingerprint) -> "BoundTest":
    def bind_count(self, fingerprint: Fingerprint, is_count: bool) -> "BoundTest":
    def bind_by_label(self, fingerprint: Fingerprint, label: Label):
    def bind_by_label_count(self, fingerprint: Fingerprint, label: Label, slot: "ByLabelCountSlot"):
    def bind_mean_std(self, fingerprint: Fingerprint, is_mean: bool = True):
    def bind_dataframe(self, fingerprint: Fingerprint, column: str, label_filters: Optional[Dict[str, str]] = None):

Import

from evidently.core.tests import GenericTest, Reference, ThresholdType, ThresholdValue, threshold_typecheck_guard

I/O Contract

Inputs

Name	Type	Required	Description
test_name	str	Yes	Name identifying the test type (e.g., "gt", "lt", "between")
metric	Optional[MetricTest]	No	MetricTest implementation for metric-level testing
descriptor	Optional[DescriptorTest]	No	DescriptorTest implementation for descriptor-level testing
fingerprint	Fingerprint	Yes (bind methods)	Metric fingerprint identifying which metric to bind the test to
label	Label	Yes (bind_by_label*)	Label value for per-label metric tests
column	str	Yes (bind_dataframe)	Column name in dataframe metric results
relative	Optional[float]	No (Reference)	Relative change threshold as a decimal (e.g., 0.1 for 10%)
absolute	Optional[float]	No (Reference)	Absolute difference threshold

Outputs

Name	Type	Description
for_metric return	MetricTest	The metric test implementation
for_descriptor return	DescriptorTest	The descriptor test implementation
bind_* return	BoundTest	A test bound to a specific metric fingerprint and result shape

Usage Examples

from evidently.core.tests import GenericTest, Reference

# Create a reference-based threshold
ref = Reference(relative=0.1, absolute=5.0)

# GenericTest is typically created internally by the framework
# and used via the bind methods during report execution:
# test.bind_single(metric_fingerprint)
# test.bind_count(metric_fingerprint, is_count=True)
# test.bind_by_label(metric_fingerprint, label="class_a")

Related Pages

Environment:Evidentlyai_Evidently_Python_Core_Environment
Evidentlyai_Evidently_Metric_Container - Containers that use include_tests to control test generation
Evidentlyai_Evidently_CLI_Report - CLI report command that displays test results

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment