Principle:Neuml Txtai Keyword Scoring
| Knowledge Sources | |
|---|---|
| Domains | Information_Retrieval, Keyword_Search |
| Last Updated | 2026-02-09 17:00 GMT |
Overview
Keyword Scoring is txtai's framework for traditional information retrieval scoring methods (TF-IDF and BM25) that provide lexical search as an alternative or complement to dense vector similarity search.
Description
While dense vector search excels at capturing semantic meaning, keyword-based scoring remains essential for queries where exact term matching matters -- product codes, proper nouns, technical identifiers, or highly specific terminology. txtai's Scoring framework provides keyword-based retrieval backends that operate alongside or independently of the ANN vector index. Two concrete implementations are available: TFIDF (an in-process scikit-learn-based TF-IDF scorer) and PGText (a PostgreSQL full-text search backend using tsvector/tsquery).
The TFIDF backend builds a sparse term-document matrix using scikit-learn's TfidfVectorizer. Each document is represented as a sparse vector where each dimension corresponds to a vocabulary term and the value is the TF-IDF weight -- the product of term frequency within the document and inverse document frequency across the corpus. At query time, the query string is converted to the same sparse representation and scored against all documents using cosine similarity. This approach is fast, memory-efficient for moderate vocabularies, and requires no external dependencies beyond scikit-learn.
The PGText backend delegates keyword scoring to PostgreSQL's built-in full-text search engine. Documents are stored with automatically generated tsvector columns (pre-processed token representations), and queries are executed as tsquery operations with PostgreSQL's ranking functions (ts_rank or ts_rank_cd). PGText is ideal for deployments already using PostgreSQL (especially those using PGVector for vector search), as it provides integrated keyword search without a separate index process. Both backends support hybrid search when combined with the dense vector index, where the final score is a weighted combination of the keyword score and the vector similarity score.
Usage
Use Keyword Scoring when queries contain specific terms that must appear in results (exact matching), when the corpus contains specialized vocabulary where semantic models may lack coverage, or when you need a sparse search baseline to complement dense vector search. Use TFIDF for self-contained deployments with no external database dependencies. Use PGText when running with PostgreSQL for a unified search infrastructure. Enable hybrid scoring (configuring the weights parameter) when both semantic understanding and lexical precision are needed.
Theoretical Basis
1. TF-IDF Weighting: Term Frequency-Inverse Document Frequency assigns each term t in document d a weight:
w(t, d) = tf(t, d) * log(N / df(t))
where tf(t, d) is the frequency of term t in document d, N is the total number of documents, and df(t) is the number of documents containing term t. This weighting promotes terms that are frequent in a document but rare across the corpus, capturing term discriminativeness.
2. BM25 Formula: BM25 (Best Matching 25) extends TF-IDF with document length normalization and term frequency saturation:
score(q, d) = SUM_{t in q} IDF(t) * (tf(t,d) * (k1 + 1)) / (tf(t,d) + k1 * (1 - b + b * |d| / avgdl))
where k1 (default 1.2) controls term frequency saturation, b (default 0.75) controls document length normalization, |d| is the document length, and avgdl is the average document length. PostgreSQL's ts_rank function implements a variant of this formula.
3. Inverted Index Structure: Both backends rely on an inverted index that maps each vocabulary term to the list of documents containing that term along with positional and frequency information. The TFIDF backend stores this as a sparse CSR (Compressed Sparse Row) matrix; the PGText backend uses PostgreSQL's GIN (Generalized Inverted Index) over tsvector columns. The inverted index enables sublinear query time proportional to the number of matching terms rather than the number of documents.
4. PostgreSQL tsvector/tsquery: PostgreSQL's full-text search represents documents as tsvector (sorted lists of lexemes with position information) and queries as tsquery (boolean expressions over lexemes). The to_tsvector() function tokenizes, stems, and removes stop words from text. The to_tsquery() function parses query syntax supporting AND (&), OR (|), NOT (!), and phrase proximity (<->). The @@ operator matches tsquery against tsvector using the GIN index.
5. Hybrid Score Fusion: When both keyword and vector scores are available, the final relevance score is computed as a weighted linear combination:
s_final(q, d) = alpha * s_vector(q, d) + (1 - alpha) * s_keyword(q, d)
where alpha in [0, 1] controls the balance. Both score components are normalized to the [0, 1] range before fusion. Setting alpha = 0.5 gives equal weight; values closer to 1.0 favor semantic search, while values closer to 0.0 favor keyword matching.