Implementation:Neuml Txtai Pipeline Init

Implementation: Pipeline_Init

Field	Value
Sources	txtai
Domains	NLP, Machine_Learning, Workflow_Orchestration
Last Updated	2026-02-10 12:00 GMT

Overview

Pipeline_Init documents the initialization constructors for the four core pipeline types -- Textractor, Summary, Translation, and LLM -- that serve as the building blocks for txtai workflow orchestration.

Description

Each pipeline constructor configures and loads the underlying AI model or processing backend. The constructors accept configuration parameters that control model selection, hardware placement (GPU/CPU), quantization, batch sizes, and backend-specific options. Once constructed, each pipeline is a callable object ready to process data.

• Textractor.__init__: Configures text extraction from files and URLs. Sets up a FileToHTML backend for document parsing, an HTMLToMarkdown converter for output formatting, and optional HTTP headers for URL retrieval. Inherits segmentation options (sentence/line/paragraph tokenization) from its parent class.

• Summary.__init__: Configures abstractive summarization by delegating to a Hugging Face Transformers summarization pipeline. The parent HFPipeline class handles model loading, GPU placement, and optional quantization.

• Translation.__init__: Configures multi-language translation with automatic language detection. Initializes the base model (defaults to facebook/m2m100_418M), prepares a model cache for dynamically loaded language-pair models, and configures language detection settings.

• LLM.__init__: Configures large language model generation. Delegates to GenerationFactory.create which selects the appropriate backend (Hugging Face Transformers, llama.cpp, LiteLLM, or custom) based on the model path and optional method parameter.

Usage

Import these classes to create pipeline instances for use in workflow tasks. Each pipeline can be used standalone or wrapped in a Task for workflow integration.

Code Reference

Source Locations

Pipeline	File	Lines
Textractor	src/python/txtai/pipeline/data/textractor.py	17--49
Summary	src/python/txtai/pipeline/text/summary.py	10--16
Translation	src/python/txtai/pipeline/text/translation.py	20--53
LLM	src/python/txtai/pipeline/llm/llm.py	15--39

Signatures

class Textractor(Segmentation):
    def __init__(
        self,
        sentences=False,
        lines=False,
        paragraphs=False,
        minlength=None,
        join=False,
        sections=False,
        cleantext=True,
        chunker=None,
        headers=None,
        backend="available",
        **kwargs
    ):

class Summary(HFPipeline):
    def __init__(self, path=None, quantize=False, gpu=True, model=None, **kwargs):

class Translation(HFModel):
    def __init__(self, path=None, quantize=False, gpu=True, batch=64, langdetect=None, findmodels=True):

class LLM(Pipeline):
    def __init__(self, path=None, method=None, **kwargs):

Import

from txtai.pipeline import Textractor, Summary, Translation, LLM

I/O Contract

Textractor.__init__ Inputs

Parameter	Type	Required	Description
sentences	bool	No	Tokenize text into sentences if True. Defaults to False.
lines	bool	No	Tokenize text into lines if True. Defaults to False.
paragraphs	bool	No	Tokenize text into paragraphs if True. Defaults to False.
minlength	int / None	No	Minimum character length per text element. Defaults to None.
join	bool	No	Join tokenized sections back together if True. Defaults to False.
sections	bool	No	Tokenize text into sections (page/section breaks) if True. Defaults to False.
cleantext	bool	No	Apply text cleaning rules. Defaults to True.
chunker	object / None	No	Third-party chunker for tokenization. Defaults to None.
headers	dict / None	No	HTTP headers for URL retrieval. Defaults to {}.
backend	str	No	File-to-HTML conversion backend. Defaults to "available". Use "tika" for Apache Tika.
**kwargs	dict	No	Additional keyword arguments passed to parent Segmentation.

Summary.__init__ Inputs

Parameter	Type	Required	Description
path	str / None	No	Model path (Hugging Face model hub id or local path). Uses default summarization model if not provided.
quantize	bool	No	Quantize model to int8 for CPU inference. Defaults to False.
gpu	bool / int	No	Enable GPU (True/False) or specify GPU device id. Defaults to True.
model	object / None	No	Existing pipeline model to wrap. Defaults to None.
**kwargs	dict	No	Additional keyword arguments passed to the Hugging Face pipeline.

Translation.__init__ Inputs

Parameter	Type	Required	Description
path	str / None	No	Model path. Defaults to "facebook/m2m100_418M" if not provided.
quantize	bool	No	Quantize model to int8. Defaults to False.
gpu	bool / int	No	Enable GPU or specify device id. Defaults to True.
batch	int	No	Batch size for incremental content processing. Defaults to 64.
langdetect	callable / str / None	No	Custom language detection function (takes list of strings, returns language codes) or model path string for default detector. Uses "neuml/language-id-quantized" if not provided.
findmodels	bool	No	Search Hugging Face Hub for source-target translation models. Defaults to True.

LLM.__init__ Inputs

Parameter	Type	Required	Description
path	str / None	No	Model path. Defaults to "ibm-granite/granite-4.0-350m" if not provided.
method	str / None	No	LLM framework to use. Infers from path if not provided. Supports Hugging Face Transformers, llama.cpp, LiteLLM, and custom implementations.
**kwargs	dict	No	Additional model keyword arguments passed to GenerationFactory.create.

Outputs

Output	Type	Description
Pipeline instance	Textractor / Summary / Translation / LLM	A configured, callable pipeline object. Call with input data to execute the transformation.

Usage Examples

Text Extraction Pipeline

from txtai.pipeline import Textractor

# Extract text from files, splitting into paragraphs
textractor = Textractor(paragraphs=True, backend="available")

# Process a local file
text = textractor("path/to/document.pdf")

# Process a URL
text = textractor("https://example.com/article.html")

Summarization Pipeline

from txtai.pipeline import Summary

# Create summarizer with default model on GPU
summary = Summary()

# Summarize text
result = summary("Long article text here...", maxlength=150)

# Summarize a batch
results = summary(["Article 1...", "Article 2..."])

Translation Pipeline

from txtai.pipeline import Translation

# Create translator
translate = Translation()

# Translate to English (auto-detects source language)
result = translate("Texto en espanol", target="en")

# Translate with explicit source language
result = translate("Bonjour le monde", target="en", source="fr")

LLM Pipeline

from txtai.pipeline import LLM

# Create LLM with a local model
llm = LLM("meta-llama/Meta-Llama-3.1-8B-Instruct")

# Generate text
result = llm("Explain quantum computing in simple terms.", maxlength=512)

# Use with chat messages
result = llm([
    {"role": "user", "content": "What is machine learning?"}
])

Related Pages

• Principle:Neuml_Txtai_Pipeline_Definition

Requires Environment

• Environment:Neuml_Txtai_Python_Core_Dependencies

Uses Heuristic

• Heuristic:Neuml_Txtai_Llama_Cpp_Context_Fallback