Implementation:Ucbepic Docetl MapOptimizer ConfigGenerators
| Knowledge Sources | |
|---|---|
| Domains | Data_Processing, Optimization |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Concrete tool for generating split, metadata, context, chunk size, and peripheral configurations for map operation optimization provided by DocETL.
Description
The ConfigGenerator class is a component of the MapOptimizer subsystem responsible for generating various configurations needed when optimizing map operations that process long documents. It determines how to split input data into chunks (identifying the split key and subprompt), checks whether metadata extraction is necessary for chunk processing, evaluates context needs (peripheral chunks, document head/tail), generates appropriate chunk size candidates based on token limits and document lengths, and produces peripheral chunk configurations with adaptive scaling based on document-to-chunk size ratios.
Usage
Use ConfigGenerator when you are building or customizing chunk-based optimization plans for map operations. It is used internally by the PlanGenerator to determine split keys, chunk sizes, metadata needs, context window requirements, and peripheral configurations. It is relevant when documents are long enough that they must be split into chunks for processing within LLM context limits.
Code Reference
Source Location
- Repository: Ucbepic_Docetl
- File: docetl/optimizers/map_optimizer/config_generators.py
- Lines: 1-541
Signature
class ConfigGenerator:
def __init__(
self,
llm_client: LLMClient,
console: Console,
config: dict[str, Any],
max_threads: int,
) -> None: ...
def _get_split_config(self, op_config: dict[str, Any], input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
def _determine_metadata_needs(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
def _check_metadata_necessity(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
def _get_metadata_config(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
def _determine_context_needs(self, op_config: dict[str, Any], subprompt: str, chunk_size: int, split_key: str, input_data_sample: list[dict[str, Any]]) -> dict[str, Any]: ...
def _generate_chunk_sizes(self, split_key: str, input_data_sample: list[dict[str, Any]], token_limit: int, num_chunks: int = 8) -> list[int]: ...
def _generate_peripheral_configs(self, summary_key: str, chunk_size: int, avg_doc_size: int) -> list[tuple[dict[str, Any], bool]]: ...
Import
from docetl.optimizers.map_optimizer.config_generators import ConfigGenerator
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| llm_client | LLMClient | Yes | Client for interacting with the language model |
| console | Console | Yes | Rich console object for output logging |
| config | dict[str, Any] | Yes | The pipeline configuration dictionary |
| max_threads | int | Yes | Maximum number of threads for parallel execution |
| op_config | dict[str, Any] | Yes | The operation configuration (for split/metadata/context methods) |
| input_data_sample | list[dict[str, Any]] | Yes | Sample of input data for analysis |
| split_key | str | Yes | The key in the input data used for splitting (for chunk/peripheral methods) |
| token_limit | int | Yes | Maximum token limit for chunk size generation |
Outputs
| Name | Type | Description |
|---|---|---|
| split_config | dict[str, Any] | Dictionary with split_key, subprompt, and subprompt_output_schema |
| metadata_config | dict[str, Any] | Dictionary with needs_metadata flag, metadata_prompt, and output_schema |
| context_config | dict[str, Any] | Dictionary with needs_peripherals, previous/next context, head/tail flags, and reason |
| chunk_sizes | list[int] | List of candidate chunk sizes to evaluate |
| peripheral_configs | list[tuple[dict[str, Any], bool]] | List of peripheral configuration tuples with summarization flags |
Usage Examples
from docetl.optimizers.map_optimizer.config_generators import ConfigGenerator
from docetl.optimizers.utils import LLMClient
config_gen = ConfigGenerator(
llm_client=llm_client,
console=console,
config=pipeline_config,
max_threads=4,
)
# Generate split configuration
split_result = config_gen._get_split_config(op_config, input_data_sample)
print(f"Split key: {split_result['split_key']}")
print(f"Subprompt: {split_result['subprompt']}")
# Generate chunk sizes
chunk_sizes = config_gen._generate_chunk_sizes(
split_key="document_text",
input_data_sample=input_data_sample,
token_limit=8192,
)
print(f"Chunk sizes to evaluate: {chunk_sizes}")
# Generate peripheral configurations
peripheral_configs = config_gen._generate_peripheral_configs(
summary_key="document_text_summary",
chunk_size=500,
avg_doc_size=5000,
)