Implementation:Run llama Llama index BaseExtractor

Overview

BaseExtractor is an abstract base class for metadata extractors in the LlamaIndex framework. It extends TransformComponent and provides the interface and common processing logic for extracting metadata from parsed document nodes. Extractors can be chained together as part of an ingestion pipeline to enrich nodes with additional metadata prior to indexing.

Source file: llama-index-core/llama_index/core/extractors/interface.py (178 lines)

Class Hierarchy

TransformComponent
  └── BaseExtractor

BaseExtractor inherits from TransformComponent, which is a Pydantic-based schema class defined in llama_index.core.schema. This makes BaseExtractor usable as a transformation step in LlamaIndex ingestion pipelines.

Configuration Fields

Field	Type	Default	Description
is_text_node_only	bool	True	Whether the extractor only processes TextNode instances
show_progress	bool	True	Whether to show progress during extraction
metadata_mode	MetadataMode	MetadataMode.ALL	Metadata mode used when reading nodes
node_text_template	str	See DEFAULT_NODE_TEXT_TEMPLATE	Template defining how node text is mixed with metadata text
disable_template_rewrite	bool	False	When True, disables rewriting of the node text template
in_place	bool	True	Whether to process nodes in place or create deep copies
num_workers	int	4	Number of workers for concurrent async processing

Default Node Text Template

The module defines a default template used to format node text alongside its metadata:

[Excerpt from document]
{metadata_str}
Excerpt:
-----
{content}
-----

This template is applied to TextNode instances during aprocess_nodes unless disable_template_rewrite is set to True.

Abstract Method

aextract

@abstractmethod
async def aextract(self, nodes: Sequence[BaseNode]) -> List[Dict]:

The core abstract method that subclasses must implement. It receives a sequence of nodes and must return a list of dictionaries, where each dictionary contains the extracted metadata for the corresponding node. This is the only method subclasses are required to override.

Synchronous and Asynchronous API

extract

def extract(self, nodes: Sequence[BaseNode]) -> List[Dict]:

A synchronous wrapper around aextract. Uses asyncio_run to execute the async method in a blocking context.

aprocess_nodes / process_nodes

async def aprocess_nodes(
    self,
    nodes: Sequence[BaseNode],
    excluded_embed_metadata_keys: Optional[List[str]] = None,
    excluded_llm_metadata_keys: Optional[List[str]] = None,
    **kwargs: Any,
) -> List[BaseNode]:

The main node processing pipeline method:

1. If in_place is True, operates on nodes directly; otherwise creates deep copies.
2. Calls aextract to get metadata dictionaries for all nodes.
3. Updates each node's metadata with the extracted key-value pairs.
4. Extends each node's excluded_embed_metadata_keys and excluded_llm_metadata_keys if provided.
5. Rewrites the text_template on TextNode instances (unless disable_template_rewrite is set).
6. Returns the modified node list.

process_nodes is the synchronous counterpart, wrapping aprocess_nodes with asyncio_run.

__call__ / acall

def __call__(self, nodes: Sequence[BaseNode], **kwargs: Any) -> List[BaseNode]:
async def acall(self, nodes: Sequence[BaseNode], **kwargs: Any) -> List[BaseNode]:

Callable interface that delegates to process_nodes / aprocess_nodes. This allows the extractor to be used directly as a callable in transformation pipelines.

Serialization

from_dict

@classmethod
def from_dict(cls, data: Dict[str, Any], **kwargs: Any) -> Self:

Class method for deserializing an extractor from a dictionary. It handles:

• Merging kwargs into the data dictionary.
• Removing the class_name key (used for type dispatch).
• Loading nested llm_predictor via load_predictor if present.
• Loading nested llm via load_llm if present.

class_name

Returns the string "MetadataExtractor" for serialization and type identification.

Dependencies

• llama_index.core.async_utils.asyncio_run -- for running async methods synchronously
• llama_index.core.schema -- provides BaseNode, TextNode, MetadataMode, TransformComponent
• llama_index.core.bridge.pydantic.Field -- Pydantic field descriptors
• typing_extensions.Self -- return type annotation for from_dict

Usage Pattern

To create a custom extractor, subclass BaseExtractor and implement aextract:

class MyCustomExtractor(BaseExtractor):
    async def aextract(self, nodes: Sequence[BaseNode]) -> List[Dict]:
        metadata_list = []
        for node in nodes:
            # Perform custom extraction logic
            metadata_list.append({"custom_key": "custom_value"})
        return metadata_list

The extractor can then be used directly in a pipeline or called on a list of nodes:

extractor = MyCustomExtractor()
enriched_nodes = extractor(nodes)