Principle:Neuml Txtai Content Indexing
| Knowledge Sources | |
|---|---|
| Domains | NLP, RAG |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Content-enabled indexing is the practice of storing original document text alongside embedding vectors in a unified index, allowing retrieval operations to return both similarity scores and the source text needed for downstream generation.
Description
Standard semantic search indexes map documents to dense vectors and return identifiers or scores at query time. For retrieval-augmented generation, however, the generative model requires the actual text of retrieved passages -- not just their identifiers. Content-enabled indexing addresses this by coupling a vector index with a document database that persists the original text of each indexed record.
When content storage is enabled, the index maintains two parallel structures: an approximate nearest neighbor (ANN) index for fast similarity search and a relational database (typically SQLite) for storing document metadata and text. At query time, the system first retrieves the top-k nearest vectors, then joins with the document database to fetch the corresponding text passages. This joined result set provides the context that a generative model uses to produce grounded answers.
The distinction between content-enabled and content-disabled indexing is critical for RAG workflows. Without content storage, the index can only return document identifiers and scores. The application would then need a separate mechanism to look up the original text, adding complexity and latency. With content enabled, the index is self-contained: it serves as both the retrieval engine and the context store for generation.
Usage
Use content-enabled indexing when:
- Building a RAG pipeline where retrieved passages must be passed directly to a generative model.
- The application needs to display source text alongside search results.
- SQL-based filtering or aggregation over document metadata is required in addition to vector search.
- The workflow requires a self-contained index that does not depend on external document stores.
Content-enabled indexing is not necessary when the sole goal is to rank documents by semantic similarity without inspecting their text (e.g., deduplication, clustering, or re-ranking workflows where text is available externally).
Theoretical Basis
Dual-Store Architecture
A content-enabled embeddings index consists of two coordinated stores:
| Component | Purpose | Data Stored |
|---|---|---|
| ANN Index | Fast approximate nearest neighbor search | Dense embedding vectors v_i for each document |
| Document Database | Content and metadata storage | Original text, document id, optional tags/metadata |
Indexing Process
Given a collection of documents D = [(id_1, text_1), (id_2, text_2), ..., (id_n, text_n)], the indexing process proceeds as follows:
1. For each document (id_i, text_i) in D:
a. Compute embedding vector: v_i = embed(text_i)
b. Insert v_i into ANN index at position i
c. Insert (id_i, text_i) into document database
2. Build ANN search structure over {v_1, ..., v_n}
Query-Time Retrieval
At query time, the system performs a two-phase lookup:
1. Compute query embedding: v_q = embed(query)
2. Retrieve top-k nearest neighbors from ANN index:
results = ANN.search(v_q, k) -> [(id_j, score_j), ...]
3. For each (id_j, score_j) in results:
text_j = database.lookup(id_j)
yield {"id": id_j, "text": text_j, "score": score_j}
This joined result provides the context passages needed by a generative model. The score can also be used to filter low-confidence matches before they enter the generation prompt.
Content Configuration
The content flag acts as a switch between two modes of operation:
- content=False (default): The index stores only vectors and a mapping from positions to external identifiers. Text must be retrieved through an external mechanism.
- content=True: The index creates and maintains a document database alongside the vector index. Search results include the original text, enabling self-contained RAG workflows.