Implementation:Ucbepic Docetl MapOptimizer ConfigGenerators

Knowledge Sources	Ucbepic_Docetl DocETL Docs
Domains	Data_Processing, Optimization
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for generating split, metadata, context, chunk size, and peripheral configurations for map operation optimization provided by DocETL.

Description

The ConfigGenerator class is a component of the MapOptimizer subsystem responsible for generating various configurations needed when optimizing map operations that process long documents. It determines how to split input data into chunks (identifying the split key and subprompt), checks whether metadata extraction is necessary for chunk processing, evaluates context needs (peripheral chunks, document head/tail), generates appropriate chunk size candidates based on token limits and document lengths, and produces peripheral chunk configurations with adaptive scaling based on document-to-chunk size ratios.

Usage

Use ConfigGenerator when you are building or customizing chunk-based optimization plans for map operations. It is used internally by the PlanGenerator to determine split keys, chunk sizes, metadata needs, context window requirements, and peripheral configurations. It is relevant when documents are long enough that they must be split into chunks for processing within LLM context limits.

Code Reference

Source Location

Repository: Ucbepic_Docetl
File: docetl/optimizers/map_optimizer/config_generators.py
Lines: 1-541

Signature

class ConfigGenerator:
    def __init__(
        self,
        llm_client: LLMClient,
        console: Console,
        config: dict[str, Any],
        max_threads: int,
    ) -> None: ...

    def _get_split_config(self, op_config: dict[str, Any], input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
    def _determine_metadata_needs(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
    def _check_metadata_necessity(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
    def _get_metadata_config(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
    def _determine_context_needs(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
    def _generate_chunk_sizes(self, split_key: str, input_data_sample: list[dict[str, Any]], token_limit: int, num_chunks: int = 8) -> list[int]: ...
    def _generate_peripheral_configs(self, summary_key: str, chunk_size: int, avg_doc_size: int) -> list[tuple[dict[str, Any], bool]]: ...

Import

from docetl.optimizers.map_optimizer.config_generators import ConfigGenerator

I/O Contract

Inputs

Name	Type	Required	Description
llm_client	LLMClient	Yes	Client for interacting with the language model
console	Console	Yes	Rich console object for output logging
config	dict[str, Any]	Yes	The pipeline configuration dictionary
max_threads	int	Yes	Maximum number of threads for parallel execution
op_config	dict[str, Any]	Yes	The operation configuration (for split/metadata/context methods)
input_data_sample	list[dict[str, Any]]	Yes	Sample of input data for analysis
split_key	str	Yes	The key in the input data used for splitting (for chunk/peripheral methods)
token_limit	int	Yes	Maximum token limit for chunk size generation

Outputs

Name	Type	Description
split_config	dict[str, Any]	Dictionary with split_key, subprompt, and subprompt_output_schema
metadata_config	dict[str, Any]	Dictionary with needs_metadata flag, metadata_prompt, and output_schema
context_config	dict[str, Any]	Dictionary with needs_peripherals, previous/next context, head/tail flags, and reason
chunk_sizes	list[int]	List of candidate chunk sizes to evaluate
peripheral_configs	list[tuple[dict[str, Any], bool]]	List of peripheral configuration tuples with summarization flags

Usage Examples

from docetl.optimizers.map_optimizer.config_generators import ConfigGenerator
from docetl.optimizers.utils import LLMClient

config_gen = ConfigGenerator(
    llm_client=llm_client,
    console=console,
    config=pipeline_config,
    max_threads=4,
)

# Generate split configuration
split_result = config_gen._get_split_config(op_config, input_data_sample)
print(f"Split key: {split_result['split_key']}")
print(f"Subprompt: {split_result['subprompt']}")

# Generate chunk sizes
chunk_sizes = config_gen._generate_chunk_sizes(
    split_key="document_text",
    input_data_sample=input_data_sample,
    token_limit=8192,
)
print(f"Chunk sizes to evaluate: {chunk_sizes}")

# Generate peripheral configurations
peripheral_configs = config_gen._generate_peripheral_configs(
    summary_key="document_text_summary",
    chunk_size=500,
    avg_doc_size=5000,
)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment