Implementation:Ucbepic Docetl MapOptimizer Evaluator
| Knowledge Sources | |
|---|---|
| Domains | Data_Processing, Optimization |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Concrete tool for evaluating and comparing optimization plan outputs using LLM-based quality assessment provided by DocETL.
Description
The Evaluator class is a component of the MapOptimizer subsystem responsible for assessing the quality of optimization plan outputs. It evaluates individual plans by running them on input data, measuring runtime, and scoring output quality using a custom validator prompt with a four-tier quality scale (Satisfactory, Mostly Satisfactory, Partially Satisfactory, Unsatisfactory). It also performs pairwise comparison of plans using an LLM judge to determine which plan produces better outputs for each sample, aggregating results into win counts for final ranking.
Usage
Use the Evaluator when you need to compare multiple candidate optimization plans for a map operation and select the best one. It is used internally by the MapOptimizer to evaluate and rank candidate plans (gleaning, chunking, parallel, chain decomposition) against each other and against the baseline no-change plan. The evaluator supports parallel plan evaluation and pairwise tournament-style comparison.
Code Reference
Source Location
- Repository: Ucbepic_Docetl
- File: docetl/optimizers/map_optimizer/evaluator.py
- Lines: 1-500
Signature
class Evaluator:
def __init__(
self,
llm_client: LLMClient,
console: Console,
run_operation: Callable[[dict[str, Any], list[dict[str, Any]]], list[dict[str, Any]]],
timeout: int = 60,
num_plans_to_evaluate_in_parallel: int = 10,
is_filter: bool = False,
) -> None: ...
def _pairwise_compare_plans(
self,
filtered_results: dict[str, tuple[float, float, list[dict[str, Any]]]],
validator_prompt: str,
op_config: dict[str, Any],
input_data: list[dict[str, Any]],
) -> dict[str, int]: ...
def _compare_two_plans(
self,
overall_prompt: str,
plan1_name: str,
plan1_output: list[dict[str, Any]],
plan2_name: str,
plan2_output: list[dict[str, Any]],
validator_prompt: str,
op_config: dict[str, Any],
input_data: list[dict[str, Any]],
) -> str | None: ...
def _evaluate_plan(
self,
plan_name: str,
op_config: dict[str, Any],
plan: dict[str, Any] | list[dict[str, Any]],
input_data: list[dict[str, Any]],
validator_prompt: str,
) -> tuple[float, float, list[dict[str, Any]]]: ...
def _assess_operation(
self,
op_config: dict[str, Any],
input_data: list[dict[str, Any]],
output_data: list[dict[str, Any]],
validator_prompt: str,
) -> dict[str, Any]: ...
def _assess_output_quality(
self,
op_config: dict[str, Any],
input_data: list[dict[str, Any]],
output_data: list[dict[str, Any]],
element_idx: int,
validator_prompt: str,
) -> str: ...
Import
from docetl.optimizers.map_optimizer.evaluator import Evaluator
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| llm_client | LLMClient | Yes | Client for interacting with the language model |
| console | Console | Yes | Rich console object for output logging |
| run_operation | Callable | Yes | Function to execute operations on input data |
| timeout | int | No | Timeout in seconds for operation execution (default: 60) |
| num_plans_to_evaluate_in_parallel | int | No | Number of plans to evaluate concurrently (default: 10) |
| is_filter | bool | No | Whether the operation is a filter operation (default: False) |
| op_config | dict[str, Any] | Yes | The operation configuration (for evaluation methods) |
| input_data | list[dict[str, Any]] | Yes | Input data to evaluate plans against |
| validator_prompt | str | Yes | Custom prompt for assessing output quality |
Outputs
| Name | Type | Description |
|---|---|---|
| average_score | float | Average quality score across all evaluated samples (1-4 scale) |
| runtime | float | Execution time of the plan in seconds |
| output_data | list[dict[str, Any]] | Output data produced by the plan |
| pairwise_rankings | dict[str, int] | Win counts from pairwise plan comparisons |
| assessment | dict[str, Any] | Operation assessment with needs_improvement flag, reasons, and improvements |
Usage Examples
from docetl.optimizers.map_optimizer.evaluator import Evaluator
evaluator = Evaluator(
llm_client=llm_client,
console=console,
run_operation=runner.run_operation,
timeout=120,
num_plans_to_evaluate_in_parallel=5,
)
# Evaluate a single plan
score, runtime, output = evaluator._evaluate_plan(
plan_name="gleaning_1_round",
op_config=op_config,
plan=gleaning_plan,
input_data=evaluation_samples,
validator_prompt=validator_prompt,
)
print(f"Plan score: {score:.2f}, runtime: {runtime:.2f}s")
# Assess the baseline operation
assessment = evaluator._assess_operation(
op_config=op_config,
input_data=input_data,
output_data=output_data,
validator_prompt=validator_prompt,
)
if assessment["needs_improvement"]:
print("Improvements needed:", assessment["improvements"])