Principle:Ucbepic Docetl Evaluation Function Registration
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Evaluation |
| Last Updated | 2026-02-08 01:40 GMT |
Overview
A quality measurement principle that enables user-defined evaluation functions to score pipeline outputs for automated optimization feedback loops.
Description
Evaluation Function Registration is the mechanism by which users provide custom scoring functions that the optimizer uses to measure pipeline quality. In the MOAR optimizer, the evaluation function is the key feedback signal that drives the Monte Carlo Tree Search: each candidate pipeline variant is scored by the evaluation function, and the Pareto frontier tracks cost vs. accuracy tradeoffs.
The evaluation function takes the original dataset file path and the pipeline results file path, then returns a dictionary of metric scores. This design allows domain-specific quality metrics (e.g., F1 score for entity extraction, exact match for classification).
Usage
Use this principle when you want to automatically optimize a DocETL pipeline. You must provide an evaluation function decorated with @register_eval that quantifies how good the pipeline output is for your specific task.
Theoretical Basis
Evaluation functions serve as the objective function in optimization:
- Define Metric: Choose a quality measure relevant to the task
- Implement Scorer: Write a function comparing outputs to ground truth or heuristic criteria
- Register: Decorate with @register_eval for automatic discovery
- Feedback Loop: Optimizer calls evaluation function to score each candidate pipeline