Implementation:Neuml Txtai Stream Call
| Knowledge Sources | |
|---|---|
| Domains | NLP, Search |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Concrete tool for normalizing diverse document formats into standard (id, data, tags) tuples provided by the txtai library.
Description
The Stream class is responsible for yielding documents as uniform (id, data, tags) tuples from an arbitrary stream of input documents. It accepts dictionaries, tuples of varying lengths, and plain strings, converting each into the canonical three-element tuple format. When an index action is specified and a document lacks an explicit ID, the class uses an AutoId generator to assign one automatically.
The __init__ method sets up the stream by capturing a reference to the parent embeddings instance and configuring the auto-ID generator based on the current offset (for upsert operations) or starting from zero (for fresh index operations). The __call__ method is a generator that iterates over the input documents, normalizes each one, optionally assigns an auto-generated ID, and yields the result. After all documents have been processed, it saves the current auto-ID sequence back to the configuration for use in subsequent operations.
Usage
The Stream class is used internally by the Embeddings.index and Embeddings.upsert methods. It is instantiated with the embeddings instance and an action type (INDEX, REINDEX, or UPSERT), and then called with the raw documents iterable. Users typically do not need to call Stream directly, but understanding its behavior is important for designing document input formats.
Code Reference
Source Location
- Repository: txtai
- File:
src/python/txtai/embeddings/index/stream.py - Lines: L9-67
Signature
class Stream:
def __init__(self, embeddings, action=None):
def __call__(self, documents):
Import
from txtai.embeddings.index import Stream
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| embeddings | Embeddings | Yes | The parent Embeddings instance. Used to access the index configuration and determine the current offset for auto-ID generation. |
| action | Action (enum) | No | The index action type (INDEX, REINDEX, or UPSERT). Controls offset behavior for auto-ID generation. When None, auto-IDs are not assigned. |
| documents | iterable | Yes | An iterable of input documents. Each element may be a dict (with optional "id" and "tags" keys), a tuple of 2 or 3 elements, or a plain string/object. |
Outputs
| Name | Type | Description |
|---|---|---|
| tuples | generator of (id, data, tags) | A generator yielding normalized three-element tuples. The id is either extracted from the input or auto-generated. The data is the original document content (string, dict, or object). Tags is either extracted from the input or set to None. |
Usage Examples
Basic Example
from txtai import Embeddings
# Documents are normalized internally via Stream
embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"})
# Plain strings - Stream assigns auto-IDs
embeddings.index(["first document", "second document", "third document"])
Tuple Input Formats
from txtai import Embeddings
embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"})
# Two-element tuples: (id, text) - Stream adds None for tags
documents = [("doc1", "first document"), ("doc2", "second document")]
embeddings.index(documents)
# Three-element tuples: (id, text, tags) - Stream passes through directly
documents = [("doc1", "first document", "tag_a"), ("doc2", "second document", "tag_b")]
embeddings.index(documents)
Dictionary Input Format
from txtai import Embeddings
embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2", "content": True})
# Dictionary input - Stream extracts "id" and "tags" keys
documents = [
{"id": "doc1", "text": "first document", "tags": "category_a"},
{"id": "doc2", "text": "second document", "tags": "category_b"}
]
embeddings.index(documents)