Principle:Run llama Llama index Pipeline Execution
| Knowledge Sources | |
|---|---|
| Domains | Data_Ingestion, RAG, Pipeline_Architecture |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
Pipeline execution is the process of running a configured ingestion pipeline to transform documents into nodes, applying each transformation sequentially with support for caching, parallel workers, and in-place or copy modes.
Description
Once an IngestionPipeline is constructed with its transformations and optional infrastructure components, the run() method (or its async counterpart arun()) executes the full pipeline. The execution process follows these steps:
- Deduplication check: If a docstore is configured, filter incoming documents against already-ingested content using the selected DocstoreStrategy
- Document-to-node conversion: Convert input Document objects into BaseNode objects
- Sequential transformation: Apply each transformation in order, passing the output nodes of one step as input to the next
- Cache lookup: Before each transformation, check the cache for previously computed results; skip the transformation if cached results exist
- Vector store insertion: If a vector store is configured, insert the final nodes automatically
- Return: Return the processed Sequence[BaseNode]
Usage
Call pipeline.run(documents=docs) after constructing the pipeline. Use show_progress=True for a progress bar, and num_workers to parallelize transformations that support it.
Theoretical Basis
Sequential Transformation Model
Each transformation in the pipeline implements the TransformComponent protocol, which accepts a list of nodes and returns a transformed list:
# Conceptual execution flow
nodes = documents_to_nodes(documents)
for transformation in pipeline.transformations:
cache_key = hash(nodes, transformation)
if cache.has(cache_key):
nodes = cache.get(cache_key)
else:
nodes = transformation(nodes)
cache.put(cache_key, nodes)
if vector_store:
vector_store.add(nodes)
return nodes
In-Place vs Copy Mode
The in_place parameter controls whether transformations modify nodes directly or operate on copies:
- in_place=True (default): Transformations modify the input nodes directly. More memory-efficient but mutates the original data.
- in_place=False: A deep copy of the nodes is made before each transformation. Safer but uses more memory.
Parallel Execution
The num_workers parameter enables parallel execution of transformations that support it. When set, the pipeline dispatches node batches to multiple worker processes, reducing wall-clock time for expensive operations like embedding generation.