Implementation:Ucbepic Docetl MapOptimizer Evaluator

Knowledge Sources	Ucbepic_Docetl DocETL Docs
Domains	Data_Processing, Optimization
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for evaluating and comparing optimization plan outputs using LLM-based quality assessment provided by DocETL.

Description

The Evaluator class is a component of the MapOptimizer subsystem responsible for assessing the quality of optimization plan outputs. It evaluates individual plans by running them on input data, measuring runtime, and scoring output quality using a custom validator prompt with a four-tier quality scale (Satisfactory, Mostly Satisfactory, Partially Satisfactory, Unsatisfactory). It also performs pairwise comparison of plans using an LLM judge to determine which plan produces better outputs for each sample, aggregating results into win counts for final ranking.

Usage

Use the Evaluator when you need to compare multiple candidate optimization plans for a map operation and select the best one. It is used internally by the MapOptimizer to evaluate and rank candidate plans (gleaning, chunking, parallel, chain decomposition) against each other and against the baseline no-change plan. The evaluator supports parallel plan evaluation and pairwise tournament-style comparison.

Code Reference

Source Location

Repository: Ucbepic_Docetl
File: docetl/optimizers/map_optimizer/evaluator.py
Lines: 1-500

Signature

class Evaluator:
    def __init__(
        self,
        llm_client: LLMClient,
        console: Console,
        run_operation: Callable[[dict[str, Any], list[dict[str, Any]]], list[dict[str, Any]]],
        timeout: int = 60,
        num_plans_to_evaluate_in_parallel: int = 10,
        is_filter: bool = False,
    ) -> None: ...

    def _pairwise_compare_plans(
        self,
        filtered_results: dict[str, tuple[float, float, list[dict[str, Any]]]],
        validator_prompt: str,
        op_config: dict[str, Any],
        input_data: list[dict[str, Any]],
    ) -> dict[str, int]: ...

    def _compare_two_plans(
        self,
        overall_prompt: str,
        plan1_name: str,
        plan1_output: list[dict[str, Any]],
        plan2_name: str,
        plan2_output: list[dict[str, Any]],
        validator_prompt: str,
        op_config: dict[str, Any],
        input_data: list[dict[str, Any]],
    ) -> str | None: ...

    def _evaluate_plan(
        self,
        plan_name: str,
        op_config: dict[str, Any],
        plan: dict[str, Any] | list[dict[str, Any]],
        input_data: list[dict[str, Any]],
        validator_prompt: str,
    ) -> tuple[float, float, list[dict[str, Any]]]: ...

    def _assess_operation(
        self,
        op_config: dict[str, Any],
        input_data: list[dict[str, Any]],
        output_data: list[dict[str, Any]],
        validator_prompt: str,
    ) -> dict[str, Any]: ...

    def _assess_output_quality(
        self,
        op_config: dict[str, Any],
        input_data: list[dict[str, Any]],
        output_data: list[dict[str, Any]],
        element_idx: int,
        validator_prompt: str,
    ) -> str: ...

Import

from docetl.optimizers.map_optimizer.evaluator import Evaluator

I/O Contract

Inputs

Name	Type	Required	Description
llm_client	LLMClient	Yes	Client for interacting with the language model
console	Console	Yes	Rich console object for output logging
run_operation	Callable	Yes	Function to execute operations on input data
timeout	int	No	Timeout in seconds for operation execution (default: 60)
num_plans_to_evaluate_in_parallel	int	No	Number of plans to evaluate concurrently (default: 10)
is_filter	bool	No	Whether the operation is a filter operation (default: False)
op_config	dict[str, Any]	Yes	The operation configuration (for evaluation methods)
input_data	list[dict[str, Any]]	Yes	Input data to evaluate plans against
validator_prompt	str	Yes	Custom prompt for assessing output quality

Outputs

Name	Type	Description
average_score	float	Average quality score across all evaluated samples (1-4 scale)
runtime	float	Execution time of the plan in seconds
output_data	list[dict[str, Any]]	Output data produced by the plan
pairwise_rankings	dict[str, int]	Win counts from pairwise plan comparisons
assessment	dict[str, Any]	Operation assessment with needs_improvement flag, reasons, and improvements

Usage Examples

from docetl.optimizers.map_optimizer.evaluator import Evaluator

evaluator = Evaluator(
    llm_client=llm_client,
    console=console,
    run_operation=runner.run_operation,
    timeout=120,
    num_plans_to_evaluate_in_parallel=5,
)

# Evaluate a single plan
score, runtime, output = evaluator._evaluate_plan(
    plan_name="gleaning_1_round",
    op_config=op_config,
    plan=gleaning_plan,
    input_data=evaluation_samples,
    validator_prompt=validator_prompt,
)
print(f"Plan score: {score:.2f}, runtime: {runtime:.2f}s")

# Assess the baseline operation
assessment = evaluator._assess_operation(
    op_config=op_config,
    input_data=input_data,
    output_data=output_data,
    validator_prompt=validator_prompt,
)
if assessment["needs_improvement"]:
    print("Improvements needed:", assessment["improvements"])

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment