Implementation:Evidentlyai Evidently Legacy Spark Base

Knowledge Sources	Evidentlyai_Evidently
Domains	ML Monitoring, Big Data, Apache Spark
Last Updated	2026-02-14 12:00 GMT

Overview

Implements the Spark engine base classes for the legacy Evidently pipeline, providing data definition creation, column type inference, and prediction column handling for PySpark DataFrames, enabling Evidently metrics to run on distributed Spark datasets.

Description

This module adapts Evidently's legacy pipeline to work with Apache Spark DataFrames instead of pandas DataFrames. It provides the essential data preprocessing layer that translates Spark data types and column semantics into Evidently's DataDefinition and ColumnDefinition models.

Type aliases:

• SparkSeries = DataFrame -- PySpark DataFrame used as a column-level abstraction.
• SparkDataFrame = DataFrame -- PySpark DataFrame for full datasets.

Internal data class:

• _InputData -- A simple dataclass holding optional reference and required current Spark DataFrames.

Key functions:

• _get_column_presence_spark -- Checks whether a column is Present, Missing, or Partially present (exists in one dataset but not the other).

• _process_column_spark -- Processes a single column name into a ColumnDefinition. Handles missing columns, partial presence (raise, skip, or keep), and type inference. Accepts an optional predefined type override.

• create_data_definition_spark -- The main entry point. Mirrors the pandas-based create_data_definition but for Spark. It:
  • Processes embedding columns from the mapping, skipping non-present columns.
  • Identifies ID, target, datetime, and prediction columns.
  • Discovers all data columns from both current and reference DataFrames.
  • Classifies columns as numerical, categorical, datetime, or text based on the mapping or auto-detection.
  • Infers the task type (classification or regression) from the target column type if not explicitly specified.
  • Collects distinct label values from the target column across both datasets.
  • Returns a fully constructed DataDefinition with all column definitions, prediction columns, task type, labels, and embeddings.

• _get_column_type_spark -- Infers the ColumnType for a column in a Spark context:
  • Checks explicit mapping overrides first.
  • Examines actual Spark data types using utility functions (is_numeric_dtype, is_string_dtype, is_integer_dtype, is_datetime64_dtype).
  • Handles type mismatches between reference and current (logs warning, uses reference type).
  • Applies cardinality-based heuristics: integer columns with few unique values (<= NUMBER_UNIQUE_AS_CATEGORICAL) are treated as categorical.
  • Has special handling for target and prediction columns, considering the task type and unique value count.

• _prediction_column -- Resolves prediction column specifications into PredictionColumns:
  • String prediction: checks presence, infers type, and maps to either predicted_values or prediction_probas based on task and type.
  • List prediction: verifies all columns are present and numerical, returns as probability columns.
  • Handles classification vs. regression disambiguation and edge cases like categorical predictions for regression.

Usage

Use this module when running Evidently legacy metrics on PySpark DataFrames. Pass the Spark engine to Report.run(engine=SparkEngine). The create_data_definition_spark function is called internally to build the data definition from Spark DataFrames.

Code Reference

Source Location

• Repository: Evidentlyai_Evidently
• File: src/evidently/legacy/spark/base.py

Signature

SparkSeries = DataFrame
SparkDataFrame = DataFrame

@dataclasses.dataclass
class _InputData:
    reference: Optional[DataFrame]
    current: DataFrame

def _get_column_presence_spark(column_name: str, data: _InputData) -> ColumnPresenceState: ...

def _process_column_spark(
    column_name: Optional[str],
    data: _InputData,
    if_partially_present: str = "raise",
    predefined_type: Optional[ColumnType] = None,
    mapping: Optional[ColumnMapping] = None,
) -> Optional[ColumnDefinition]: ...

def create_data_definition_spark(
    reference_data: Optional[DataFrame],
    current_data: DataFrame,
    mapping: ColumnMapping,
) -> DataDefinition: ...

def _get_column_type_spark(
    column_name: str, data: _InputData, mapping: Optional[ColumnMapping] = None
) -> ColumnType: ...

def _prediction_column(
    prediction: Optional[Union[str, int, Sequence[int], Sequence[str]]],
    target_type: Optional[ColumnType],
    task: Optional[str],
    data: _InputData,
    mapping: Optional[ColumnMapping] = None,
) -> Optional[PredictionColumns]: ...

Import

from evidently.legacy.spark.base import (
    create_data_definition_spark,
    SparkSeries,
    SparkDataFrame,
)

I/O Contract

Inputs

Name	Type	Required	Description
reference_data	Optional[pyspark.sql.DataFrame]	No	Reference/baseline Spark DataFrame.
current_data	pyspark.sql.DataFrame	Yes	Current/production Spark DataFrame.
mapping	ColumnMapping	Yes	Column mapping specifying target, prediction, feature roles, embeddings, and task type.
column_name	str	Yes (for column functions)	Name of the column to process or type-check.
if_partially_present	str	No	Strategy for partially present columns: "raise" (default), "skip", or "keep".
predefined_type	Optional[ColumnType]	No	Override for automatic column type detection.
prediction	Optional[Union[str, int, Sequence]]	No (for _prediction_column)	Prediction column specification from the mapping.
target_type	Optional[ColumnType]	No (for _prediction_column)	The inferred type of the target column.
task	Optional[str]	No	Task type: "classification", "regression", or None for auto-detection.

Outputs

Name	Type	Description
DataDefinition	DataDefinition	Complete data schema definition for the Spark datasets, including all column definitions, task type, and labels.
ColumnDefinition	Optional[ColumnDefinition]	Column name and inferred type for a single column.
ColumnType	ColumnType	Inferred column type (Numerical, Categorical, Datetime, Text).
ColumnPresenceState	ColumnPresenceState	Column presence status (Present, Missing, Partially).
PredictionColumns	Optional[PredictionColumns]	Resolved prediction column structure.

Usage Examples

from pyspark.sql import SparkSession
from evidently.legacy.spark.base import create_data_definition_spark
from evidently.legacy.pipeline.column_mapping import ColumnMapping

spark = SparkSession.builder.appName("evidently_example").getOrCreate()

# Create Spark DataFrames
reference_df = spark.createDataFrame([
    (1, 25, "M", 0),
    (2, 30, "F", 1),
    (3, 35, "M", 0),
], ["id", "age", "gender", "target"])

current_df = spark.createDataFrame([
    (4, 28, "F", 1),
    (5, 40, "M", 0),
    (6, 22, "F", 1),
], ["id", "age", "gender", "target"])

mapping = ColumnMapping(
    target="target",
    numerical_features=["age"],
    categorical_features=["gender"],
    id="id",
)

# Build data definition from Spark DataFrames
data_def = create_data_definition_spark(reference_df, current_df, mapping)

# Inspect the definition
for col in data_def.get_columns():
    print(f"{col.column_name}: {col.column_type}")

# Use with Report (via Spark engine)
# from evidently.legacy.report.report import Report
# from evidently.legacy.spark.engine import SparkEngine
# report = Report(metrics=[...])
# report.run(reference_data=reference_df, current_data=current_df,
#            column_mapping=mapping, engine=SparkEngine)

Related Pages

• Environment:Evidentlyai_Evidently_Python_Core_Environment
• Evidentlyai_Evidently_Legacy_Base_Metric -- Base Metric and InputData classes (pandas variant)
• Evidentlyai_Evidently_Legacy_Report -- Report class that can use the Spark engine
• Evidentlyai_Evidently_Legacy_Data_Drift_Calculations -- Drift calculations that consume DataDefinition
• Evidentlyai_Evidently_Legacy_Data_Quality_Calculations -- Quality calculations that consume DataDefinition