Principle:Wandb Weave Evaluation Execution

Knowledge Sources	Weave Docs Wandb Weave
Domains	Evaluation, LLM_Operations
Last Updated	2026-02-14 00:00 GMT

Overview

An orchestration pattern that systematically applies a model to a dataset and runs scorers to produce evaluation results.

Description

Evaluation Execution is the core loop that drives model evaluation. It iterates over dataset examples, applies the model to each (optionally with input preprocessing), runs all configured scorers on the model output, and aggregates results into summary statistics. The entire process is traced, producing a hierarchical call tree with the evaluation as the root.

Usage

Use this principle when you have a prepared dataset, a model, and scorers ready for evaluation. It is the central step that ties the evaluation pipeline together.

Theoretical Basis

The evaluation loop follows a map-score-reduce pattern:

1. Map: For each dataset example, apply the model to produce a prediction.
2. Score: For each prediction, apply all scorers to produce per-example scores.
3. Reduce: Aggregate per-example scores into summary statistics using auto_summarize.

The process supports:

• Parallel execution: Multiple examples are processed concurrently (controlled by WEAVE_PARALLELISM env var).
• Multiple trials: The same example can be evaluated multiple times to measure variance.
• Input preprocessing: A transform function can modify dataset rows before passing them to the model.

Related Pages

Implemented By

• Implementation:Wandb_Weave_Evaluation