Implementation:Evidentlyai Evidently Legacy Base Suite

Knowledge Sources	Evidentlyai_Evidently
Domains	Testing, Metrics, Pipeline, Reporting
Last Updated	2026-02-14 12:00 GMT

Overview

Provides the foundational infrastructure for Evidently's legacy test suite and report pipeline, including execution context management, metric/test lifecycle orchestration, snapshot serialization, and HTML dashboard rendering.

Description

The base_suite module defines the core execution pipeline for Evidently's legacy metric and test system. It contains several interrelated components:

State Machine: The pipeline follows a state machine pattern with states defined in the States class: Init, AdditionalFeatures, Verified, Calculated, and Tested. These states govern what operations are permitted at each stage of the pipeline lifecycle.

Context and ContextPayload: The Context dataclass tracks the full execution state of a pipeline run, including the calculation engine, metrics, tests, their results, generated features, data definitions, and run metadata. ContextPayload is a serializable Pydantic model that captures the context for persistence as a snapshot.

Suite: The Suite class manages the ordered execution of metrics and tests. It handles dependency discovery between metrics and tests via _discover_dependencies(), delegates calculation to the configured engine, and runs test checks after calculation completes. Tests that are non-critical and fail are downgraded to WARNING status.

Display: The Display abstract base class provides HTML rendering capabilities including show() for Jupyter notebooks, save_html() for file export (single-file or multi-file mode), and JSON serialization via json() and save_json().

Snapshot: The Snapshot model is the persistence unit for suite results. It stores a ContextPayload along with metadata, tags, timestamps, and dataset links. Snapshots can be saved/loaded from JSON files and can be materialized back into Report or TestSuite objects.

ReportBase: The ReportBase class combines Display and Runnable to form the base for Report and TestSuite implementations. It manages the inner suite, snapshot creation, column mapping extraction, and descriptor tracking.

Usage

This module is used internally as the base infrastructure for evidently.legacy.report.Report and evidently.legacy.test_suite.TestSuite. Developers building custom report or test suite implementations should extend ReportBase. The Snapshot class is used for persisting and loading pipeline results in the legacy workspace system.

Code Reference

Source Location

Repository: Evidentlyai_Evidently
File: src/evidently/legacy/suite/base_suite.py

Signature

@dataclasses.dataclass
class State:
    name: str

class States:
    Init = State("Init")
    AdditionalFeatures = State("AdditionalFeatures")
    Verified = State("Verified")
    Calculated = State("Calculated")
    Tested = State("Tested")

@dataclasses.dataclass
class Context:
    engine: Optional[Engine]
    metrics: list
    tests: list
    metric_results: Dict[Metric, Union[MetricResult, ErrorResult]]
    test_results: Dict[Test, TestResult]
    state: State
    renderers: RenderersDefinitions
    data: Optional[GenericInputData] = None
    features: Optional[Dict[GeneratedFeatures, FeatureResult]] = None
    options: Options = Options()
    data_definition: Optional[DataDefinition] = None
    run_metadata: RunMetadata = dataclasses.field(default_factory=RunMetadata)

class ContextPayload(BaseModel):
    metrics: List[Metric]
    metric_results: List[Union[MetricResult, ErrorResult]]
    tests: List[Test]
    test_results: List[TestResult]
    options: Options = Options()
    data_definition: Optional[DataDefinition]
    run_metadata: RunMetadata = RunMetadata()

class Display:
    def show(self, mode="auto") -> HTML: ...
    def save_html(self, filename, mode=SaveMode.SINGLE_FILE): ...
    def json(self, include_render=False, ...) -> str: ...
    def save_json(self, filename, ...): ...

class Suite:
    context: Context
    def __init__(self, options: Options): ...
    def add_metric(self, metric: Metric): ...
    def add_test(self, test: Test): ...
    def verify(self): ...
    def run_calculate(self, data: GenericInputData): ...
    def run_checks(self): ...
    def raise_for_error(self): ...
    def reset(self): ...

class Snapshot(BaseModel):
    id: SnapshotID
    name: Optional[str]
    timestamp: datetime
    metadata: Dict[str, MetadataValueType]
    tags: List[str]
    suite: ContextPayload
    metrics_ids: List[int]
    test_ids: List[int]
    options: Options
    links: SnapshotLinks

class ReportBase(Display, Runnable):
    _inner_suite: Suite
    options: Options
    id: SnapshotID
    timestamp: datetime

Import

from evidently.legacy.suite.base_suite import Suite
from evidently.legacy.suite.base_suite import Context, ContextPayload
from evidently.legacy.suite.base_suite import Snapshot, SnapshotLinks
from evidently.legacy.suite.base_suite import Display, ReportBase, Runnable
from evidently.legacy.suite.base_suite import States, State
from evidently.legacy.suite.base_suite import DatasetLinks, DatasetInputOutputLinks
from evidently.legacy.suite.base_suite import RunMetadata
from evidently.legacy.suite.base_suite import ExecutionError

I/O Contract

Suite.run_calculate

Parameter	Type	Description
data	`GenericInputData`	Input data containing current and optional reference datasets

Output	Type	Description
(side effect)	`Context.metric_results`	Populated metric results dictionary
(side effect)	`Context.state`	Set to `States.Calculated`

Suite.run_checks

Precondition	Description
State must be Calculated	Raises `ExecutionError` if state is Init or Verified

Output	Type	Description
(side effect)	`Context.test_results`	Populated test results dictionary
(side effect)	`Context.state`	Set to `States.Tested`

Snapshot I/O

Method	Input	Output
save(filename)	`str` (file path)	JSON file on disk
load(filename)	`str` (file path)	`Snapshot` instance
as_report()	(none)	`Report` instance
as_test_suite()	(none)	`TestSuite` instance

Usage Examples

from evidently.legacy.suite.base_suite import Suite, Snapshot
from evidently.legacy.options.base import Options

# Create and run a suite
suite = Suite(options=Options())
suite.set_engine(engine)
suite.add_metric(my_metric)
suite.add_test(my_test)

suite.run_calculate(input_data)
suite.run_checks()
suite.raise_for_error()

# Save and load snapshots
snapshot = Snapshot.load("my_snapshot.json")
report = snapshot.as_report()
report.show()

snapshot.save("output_snapshot.json")

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment