Implementation:Neuml Txtai PgText Scoring

Knowledge Sources	Neuml_Txtai
Domains	Information Retrieval, Scoring, PostgreSQL, Full-Text Search
Last Updated	2026-02-10 01:00 GMT

Overview

Concrete tool for PostgreSQL full-text search (FTS) scoring provided by txtai.

Description

The PGText class extends the base Scoring class to implement full-text search backed by PostgreSQL's built-in tsvector and ts_rank capabilities. It uses SQLAlchemy to manage database connections, sessions, and schema.

The implementation creates a table with three columns: indexid (integer primary key), text (text content), and vector (a computed TSVECTOR column generated via to_tsvector(language, text)). A GIN index is created on the vector column for efficient full-text search lookups.

Search queries are ranked using PostgreSQL's ts_rank function with plainto_tsquery for query parsing. Wildcard expressions are supported by converting bare asterisks to the PostgreSQL prefix wildcard syntax (:*). Results with scores below 1e-5 are filtered out.

Key characteristics:

Database-backed: All data resides in PostgreSQL; no in-memory index is maintained.
Language-configurable: The text search language defaults to "english" but can be set via the language config key.
Schema support: Optional PostgreSQL schema isolation via the schema config key.
Session management: Uses SQLAlchemy sessions with explicit commit/rollback for save/load operations.
Always sparse and normalized: Both issparse() and isnormalized() return True.

Usage

Use PGText when you need full-text search backed by a PostgreSQL database rather than an in-memory index. This is ideal for large-scale deployments where you want to leverage PostgreSQL's mature full-text search capabilities, need persistent storage without file-based serialization, or want to integrate with an existing PostgreSQL infrastructure.

Code Reference

Source Location

Repository: Neuml_Txtai
File: src/python/txtai/scoring/pgtext.py

Signature

class PGText(Scoring):
    def __init__(self, config=None)
    def insert(self, documents, index=None, checkpoint=None)
    def delete(self, ids)
    def weights(self, tokens) -> None
    def search(self, query, limit=3) -> list
    def batchsearch(self, queries, limit=3, threads=True) -> list
    def count(self) -> int
    def load(self, path)
    def save(self, path)
    def close(self)
    def issparse(self) -> bool
    def isnormalized(self) -> bool
    def initialize(self, recreate=False)
    def sqldialect(self, sql, parameters=None)

Import

from txtai.scoring import PGText

I/O Contract

Inputs

Name	Type	Required	Description
config	dict	No	Configuration dictionary. Supports keys: url (str, PostgreSQL connection URL; falls back to SCORING_URL env var), language (str, default "english"), schema (str, optional PostgreSQL schema), table (str, default "scoring").
documents	iterable	Yes (insert)	Iterable of (uid, document, tags) tuples. Document can be str, list, or dict.
index	int	No	Starting index id offset for document insertion.
ids	list	Yes (delete)	List of index IDs to remove from the scoring table.
query	str	Yes (search)	Full-text search query string. Supports wildcard (*) expressions.
limit	int	No	Maximum number of search results (default 3).
path	str	Yes (load/save)	Path parameter (used for session management, not file I/O).

Outputs

Name	Type	Description
search results	list[tuple(int, float)]	List of (indexid, ts_rank_score) tuples sorted by descending rank, filtered to scores > 1e-5.
count	int	Number of rows in the scoring table.
weights	None	Token weights are not supported by PGText.

Usage Examples

from txtai.scoring import PGText

# Create PGText scoring with PostgreSQL connection
scoring = PGText({
    "url": "postgresql://user:pass@localhost/mydb",
    "language": "english",
    "table": "search_index"
})

# Insert documents
documents = [
    (0, "PostgreSQL full text search capabilities", None),
    (1, "Building search engines with Python", None),
    (2, "Database indexing and query optimization", None),
]

scoring.insert(documents)
scoring.save("/tmp/pgtext")

# Search using PostgreSQL ts_rank
results = scoring.search("search engines", limit=10)
# Returns: [(1, 0.075), (0, 0.061)] (approximate ts_rank scores)

# Batch search (sequential, no threading)
queries = ["full text search", "database indexing"]
results = scoring.batchsearch(queries, limit=5)

# Cleanup
scoring.close()

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment