Implementation:Ucbepic Docetl Directive ReduceGleaning
| Knowledge Sources | |
|---|---|
| Domains | Pipeline_Optimization, LLM_Operations |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Concrete tool for adding a validation loop to Reduce operations with a judge LLM provided by the DocETL reasoning optimizer.
Description
The ReduceGleaningDirective class adds a validation loop to Reduce operations: after each LLM generation during the reduce process, a separate "judge" LLM evaluates the output using a yes/no validation prompt. If the output fails, the original LLM refines its answer and repeats until the output passes or the max number of rounds is reached. This ensures completeness and accuracy for complex aggregation tasks.
Usage
The MOAR agent applies this directive when reduce operations process complex documents that require comprehensive analysis and synthesis (e.g., research paper analysis, customer feedback consolidation, legal document review, literature synthesis) where outputs must be validated for completeness, accuracy, and proper coverage of all input materials.
Code Reference
Source Location
- Repository: Ucbepic_Docetl
- File: docetl/reasoning_optimizer/directives/reduce_gleaning.py
- Lines: 1-343
Signature
class ReduceGleaningDirective(Directive):
name = "reduce_gleaning"
description = "Adds a validation loop to Reduce operations with a judge LLM for iterative refinement."
def check_applicability(self, ...) -> Tuple[bool, str]: ...
def apply(self, ...) -> Tuple[List[Dict], List[Dict], str, dict]: ...
Import
from docetl.reasoning_optimizer.directives.reduce_gleaning import ReduceGleaningDirective
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| op_config | Dict | Yes | Operation configuration to transform |
| pipeline_ops | List[Dict] | Yes | Full pipeline operations list |
| op_idx | int | Yes | Index of target operation |
| dataset_descriptions | Dict | Yes | Dataset schema descriptions |
Outputs
| Name | Type | Description |
|---|---|---|
| new_ops | List[Dict] | Transformed operation configs |
| new_steps | List[Dict] | Updated pipeline steps |
| explanation | str | Human-readable description of changes |
| metadata | dict | Additional metadata about the transformation |
Usage Examples
# Directives are typically invoked by the MOAR agent automatically
# Example of manual invocation:
from docetl.reasoning_optimizer.directives.reduce_gleaning import ReduceGleaningDirective
directive = ReduceGleaningDirective()
applicable, reason = directive.check_applicability(op_config, pipeline_ops, op_idx, dataset_descriptions)
if applicable:
new_ops, new_steps, explanation, metadata = directive.apply(op_config, pipeline_ops, op_idx, dataset_descriptions)