Implementation:Ucbepic Docetl Directive MapReduceFusion
| Knowledge Sources | |
|---|---|
| Domains | Pipeline_Optimization, LLM_Operations |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Concrete tool for fusing a Map and Reduce operation by updating the Map to pre-extract information for the Reduce provided by the DocETL reasoning optimizer.
Description
The MapReduceFusionDirective class transforms a Map operation followed by a Reduce operation over long documents by updating the Map prompt to first extract or process only the relevant information from each document. Then, it modifies the Reduce prompt to operate on these processed outputs instead of the full document content, making the Reduce operation more efficient and focused.
Usage
The MOAR agent applies this directive when there is a Map operation followed by a Reduce operation, and the Reduce step iterates over long documents to extract specific information (e.g., locations, entities, themes) that could be pre-extracted per document in the Map step. Both the Map and Reduce operators must be specified as target operators.
Code Reference
Source Location
- Repository: Ucbepic_Docetl
- File: docetl/reasoning_optimizer/directives/map_reduce_fusion.py
- Lines: 1-387
Signature
class MapReduceFusionDirective(Directive):
name = "map_reduce_fusion"
description = "Transform Map -> Reduce by updating the Map prompt to pre-extract information for the Reduce."
def check_applicability(self, ...) -> Tuple[bool, str]: ...
def apply(self, ...) -> Tuple[List[Dict], List[Dict], str, dict]: ...
Import
from docetl.reasoning_optimizer.directives.map_reduce_fusion import MapReduceFusionDirective
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| op_config | Dict | Yes | Operation configuration to transform |
| pipeline_ops | List[Dict] | Yes | Full pipeline operations list |
| op_idx | int | Yes | Index of target operation |
| dataset_descriptions | Dict | Yes | Dataset schema descriptions |
Outputs
| Name | Type | Description |
|---|---|---|
| new_ops | List[Dict] | Transformed operation configs |
| new_steps | List[Dict] | Updated pipeline steps |
| explanation | str | Human-readable description of changes |
| metadata | dict | Additional metadata about the transformation |
Usage Examples
# Directives are typically invoked by the MOAR agent automatically
# Example of manual invocation:
from docetl.reasoning_optimizer.directives.map_reduce_fusion import MapReduceFusionDirective
directive = MapReduceFusionDirective()
applicable, reason = directive.check_applicability(op_config, pipeline_ops, op_idx, dataset_descriptions)
if applicable:
new_ops, new_steps, explanation, metadata = directive.apply(op_config, pipeline_ops, op_idx, dataset_descriptions)