Implementation:Ucbepic Docetl OpContainers

Knowledge Sources	Ucbepic_Docetl
Domains	Data_Processing, Pipeline_Execution
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for implementing a pull-based execution DAG with lazily-evaluated operation containers provided by DocETL.

Description

The containers module defines the OpContainer and StepBoundary classes that form the nodes of an execution DAG used by the DSLRunner. Each container wraps a single pipeline operation and implements a pull-based execution model: operations are only evaluated when their output is requested by a parent node, flowing backwards from the final output through children until reaching leaf scan nodes. The module supports transparent caching via checkpoints, cost tracking, selectivity estimation, and integrated optimization through MapOptimizer, ReduceOptimizer, and JoinOptimizer.

Usage

Use these containers when building or extending the DocETL pipeline execution engine. They are instantiated internally by DSLRunner when constructing the query plan from a YAML pipeline configuration.

Code Reference

Source Location

• Repository: Ucbepic_Docetl
• File: docetl/containers.py
• Lines: 1-618

Signature

class OpContainer:
    def __init__(self, name: str, runner: "DSLRunner", config: dict, **kwargs): ...
    def to_string(self) -> str: ...
    def add_child(self, child: "OpContainer") -> None: ...
    def optimize(self): ...
    def next(self, is_build: bool = False, sample_size_needed: int = None) -> tuple[list[dict], float, str]: ...
    def syntax_check(self) -> str: ...

class StepBoundary(OpContainer):
    def next(self, is_build: bool = False, sample_size_needed: int = None) -> tuple[list[dict], float, str]: ...
    def syntax_check(self) -> str: ...

Import

from docetl.containers import OpContainer, StepBoundary

I/O Contract

Inputs

Name	Type	Required	Description
name	str	Yes	Unique name for the operation container (e.g., "step_name/op_name")
runner	DSLRunner	Yes	Reference to the parent DSLRunner instance
config	dict	Yes	Operation configuration dictionary with keys like "type", "name", "prompt", etc.
selectivity	float	No	Estimated selectivity ratio for sampling calculations
left_name	str	No	Left dataset name (for equijoin operations)
right_name	str	No	Right dataset name (for equijoin operations)

Outputs

Name	Type	Description
output_data	list[dict]	The operation's output documents
cost	float	Total execution cost (this operation plus children)
logs	str	Rich-formatted execution log string

Usage Examples

# OpContainers are created internally by DSLRunner during query plan construction.
# Example of how the pull-based execution works:

from docetl.containers import OpContainer

# The runner creates a DAG of containers:
scan_container = OpContainer("step1/scan_input", runner, {"type": "scan", "dataset_name": "my_data"})
map_container = OpContainer("step1/extract_info", runner, {"type": "map", "name": "extract_info", "prompt": "..."})
map_container.add_child(scan_container)

# Execution flows backwards - requesting data from the final node:
output_data, cost, logs = map_container.next()

Related Pages

• Environment:Ucbepic_Docetl_Python_Runtime