Implementation:Ucbepic Docetl ExperimentEvalUtils

Knowledge Sources	Ucbepic_Docetl
Domains	Data_Processing, Evaluation, Experimentation
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for central evaluation orchestration across reasoning experiment datasets provided by DocETL.

Description

The evaluation utils module provides the central orchestration layer for evaluating experiment results across multiple datasets (CUAD, BlackVault, Game Reviews, MedEC, Sustainability, BioDEx, Facility). It imports dataset-specific evaluation functions, maintains a mapping of dataset names to accuracy metrics (dataset_accuracy_metrics), and provides functions for Pareto frontier identification (identify_pareto_frontier), frontier summary printing (print_pareto_frontier_summary), frontier result saving (save_pareto_frontier_results), evaluation function dispatch (get_evaluate_func), dataset statistics retrieval (get_dataset_stats), and full dataset evaluation runs (run_dataset_evaluation). Internal helpers extract node data, compute display paths, and process evaluation items generically.

Usage

Use this module when running experiments to evaluate optimized pipelines against ground truth. It is the primary interface for the MOAR experiment runner and simple agent to evaluate their pipeline outputs.

Code Reference

Source Location

• Repository: Ucbepic_Docetl
• File: experiments/reasoning/evaluation/utils.py
• Lines: 1-743

Signature

dataset_accuracy_metrics: dict  # Maps dataset names to primary metric keys

def _extract_node_data(item) -> tuple: ...
def _get_display_path(jf, output_path) -> str: ...
def _add_frontier_info(result, item) -> dict: ...
def _process_evaluation_items(nodes_or_files, evaluate_func, output_path, method_name, result_fields, field_mapping) -> list: ...

def identify_pareto_frontier(eval_results: list, dataset: str, custom_metric_key: str = None) -> list: ...
def print_pareto_frontier_summary(eval_results: list, dataset: str, custom_metric_key: str = None): ...
def save_pareto_frontier_results(eval_results: list, dataset: str, output_path, custom_metric_key: str = None): ...

def get_evaluate_func(dataset: str, mode: str = "train", custom_evaluate_func=None) -> callable: ...
def get_dataset_stats(dataset: str, yaml_path: str) -> str: ...

def run_dataset_evaluation(dataset, nodes_or_files, output_path, ground_truth_path=None, method_name="docetl", root_cost=None, custom_evaluate_func=None, custom_metric_key=None) -> tuple[list, float]: ...

Import

from experiments.reasoning.evaluation.utils import (
    run_dataset_evaluation,
    get_evaluate_func,
    dataset_accuracy_metrics,
    identify_pareto_frontier,
)

I/O Contract

Inputs

Name	Type	Required	Description
dataset	str	Yes	Dataset name (e.g., "cuad", "blackvault", "game_reviews", "medec")
nodes_or_files	list	Yes	List of Node objects or file path strings/dicts to evaluate
output_path	Path	Yes	Directory for saving evaluation results
ground_truth_path	str or None	No	Path to ground truth file (uses default if None)
method_name	str	No	Name of the method being evaluated (default: "docetl")
mode	str	No	"train" or "test" mode for evaluation function dispatch
custom_evaluate_func	callable or None	No	Custom evaluation function overriding dataset defaults
custom_metric_key	str or None	No	Custom accuracy metric key overriding dataset-specific defaults

Outputs

Name	Type	Description
eval_results	list[dict]	List of evaluation result dicts with metrics, cost, and frontier status
pareto_auc	float	Area under the Pareto frontier curve (normalized)
evaluate_func	callable	Dataset-specific evaluation function

Usage Examples

from experiments.reasoning.evaluation.utils import run_dataset_evaluation, get_evaluate_func
from pathlib import Path

# Get the evaluation function for CUAD dataset
eval_func = get_evaluate_func("cuad", mode="train")

# Run evaluation on a set of result files
eval_results, pareto_auc = run_dataset_evaluation(
    dataset="cuad",
    nodes_or_files=["output/plan_1.json", "output/plan_2.json"],
    output_path=Path("experiment_results/"),
    method_name="moar_optimizer",
)

for result in eval_results:
    print(f"File: {result['file']}, F1: {result.get('f1')}, Cost: {result['cost']}")

Related Pages

• Environment:Ucbepic_Docetl_Python_Runtime