Implementation:Neuml Txtai BM25 Scoring

Knowledge Sources	Neuml_Txtai
Domains	Information Retrieval, Scoring, Text Search
Last Updated	2026-02-10 01:00 GMT

Overview

Concrete tool for Best Matching 25 (BM25) scoring provided by txtai.

Description

The BM25 class extends TFIDF to implement the Okapi BM25 ranking function, a probabilistic information retrieval scoring method widely considered an improvement over basic TF-IDF.

It overrides two key methods from TFIDF:

computeidf: Computes BM25 IDF as log(1 + (total - freq + 0.5) / (freq + 0.5)), which is the standard BM25 IDF formula that penalizes terms appearing in more than half the corpus.
score: Computes the BM25 score as idf * (freq * (k1 + 1)) / (freq + k) where k = k1 * ((1 - b) + b * length / avgdl). The k1 parameter controls term frequency saturation (default 1.2) and b controls document length normalization (default 0.75).

All other functionality including document insertion, deletion, terms index management, batch search, content storage, and serialization is inherited from TFIDF.

Usage

Use BM25 when you need state-of-the-art keyword scoring that handles term frequency saturation and document length normalization better than basic TF-IDF. BM25 is the standard scoring method for most modern search applications and is the recommended choice for keyword-based text search in txtai.

Code Reference

Source Location

Repository: Neuml_Txtai
File: src/python/txtai/scoring/bm25.py

Signature

class BM25(TFIDF):
    def __init__(self, config=None)
    def computeidf(self, freq) -> ndarray
    def score(self, freq, idf, length) -> ndarray

Import

from txtai.scoring import BM25

I/O Contract

Inputs

Name	Type	Required	Description
config	dict	No	Configuration dictionary. Inherits all TFIDF config keys. Additionally supports: k1 (float, default 1.2) for term frequency saturation control and b (float, default 0.75) for document length normalization.
freq	ndarray	Yes (score)	Term frequency array.
idf	float or ndarray	Yes (score)	IDF score(s) for the term(s).
length	int or ndarray	Yes (score)	Document length(s) in tokens.

Outputs

Name	Type	Description
idf scores	ndarray	BM25 IDF scores computed from document frequency array.
scores	ndarray	BM25 relevance scores incorporating term frequency saturation and length normalization.

Usage Examples

from txtai.scoring import BM25

# Create BM25 scoring with custom parameters
scoring = BM25({
    "k1": 1.5,
    "b": 0.8,
    "terms": {},
    "normalize": True
})

# Insert documents
documents = [
    (0, "machine learning algorithms for classification", None),
    (1, "deep learning neural network architectures", None),
    (2, "reinforcement learning in game environments", None),
]

scoring.insert(documents)
scoring.index()

# Search using BM25 ranking
results = scoring.search("learning algorithms", limit=10)
# Returns: [(0, 0.92), (1, 0.45), (2, 0.40)] (approximate scores)

# Save and reload
scoring.save("/tmp/bm25_index")
scoring.close()

loaded = BM25({"k1": 1.5, "b": 0.8, "terms": {}, "normalize": True})
loaded.load("/tmp/bm25_index")

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment