Implementation:Recommenders team Recommenders TfidfRecommender

Knowledge Sources	Recommenders
Domains	Content-Based Filtering, Natural Language Processing, Recommendation Systems
Last Updated	2026-02-10 00:00 GMT

Overview

The TfidfRecommender class implements a content-based recommendation system using TF-IDF vectorization combined with cosine similarity to find and rank similar items based on their textual content.

Description

The TfidfRecommender provides a complete pipeline for content-based recommendations built on scikit-learn's TfidfVectorizer and cosine similarity via linear_kernel. The pipeline consists of five stages:

1. Text Cleaning -- The clean_dataframe method preprocesses text by removing HTML tags, special characters, newlines, and normalizing unicode. Text is lowercased unless BERT tokenization is selected.
2. Tokenization -- The tokenize_text method supports four tokenization methods: "none" (plain text), "nltk" (Porter stemming via NLTK), "bert" (BERT-base-cased via HuggingFace), and "scibert" (SciBERT via AllenAI). All methods feed into scikit-learn's TfidfVectorizer with configurable n-gram ranges and minimum document frequency.
3. Fitting -- The fit method computes the TF-IDF matrix using fit_transform on the tokenized text.
4. Recommendation -- The recommend_top_k_items method computes pairwise cosine similarities using linear_kernel, sorts items by similarity score, and returns the top-k recommendations per item in a tabular DataFrame.
5. Result Enrichment -- The get_top_k_recommendations method enriches results with metadata columns and optionally renders clickable URLs.

This module is particularly useful for cold-start scenarios or document/paper recommendation use cases where item text features (such as abstracts and full text) are available.

Usage

Use TfidfRecommender when you need content-based recommendations driven by textual similarity rather than collaborative filtering signals. It is well-suited for recommending scientific papers, articles, or any items with rich text descriptions, especially when user interaction data is sparse or unavailable.

Code Reference

Source Location

• Repository: Recommenders
• File: recommenders/models/tfidf/tfidf_utils.py
• Lines: 1-397

Signature

class TfidfRecommender:
    def __init__(self, id_col, tokenization_method="scibert")
    def clean_dataframe(self, df, cols_to_clean, new_col_name="cleaned_text")
    def tokenize_text(self, df_clean, text_col="cleaned_text", ngram_range=(1, 3), min_df=0.0)
    def fit(self, tf, vectors_tokenized)
    def get_tokens(self)
    def get_stop_words(self)
    def recommend_top_k_items(self, df_clean, k=5)
    def get_top_k_recommendations(self, metadata, query_id, cols_to_keep=[], verbose=True)

Import

from recommenders.models.tfidf.tfidf_utils import TfidfRecommender

I/O Contract

Inputs

Name	Type	Required	Description
id_col	str	Yes	Name of the column containing item IDs in the DataFrame
tokenization_method	str	No	Tokenization method to use: "none", "nltk", "bert", or "scibert" (default: "scibert")
df	pandas.DataFrame	Yes	DataFrame containing the text content to clean (for clean_dataframe)
cols_to_clean	list of str	Yes	Column names containing text to clean and concatenate (for clean_dataframe)
df_clean	pandas.DataFrame	Yes	DataFrame with cleaned text (for tokenize_text, fit, recommend_top_k_items)
ngram_range	tuple of int	No	Lower and upper boundary for n-gram extraction (default: (1, 3))
min_df	float	No	Minimum document frequency threshold (default: 0.0)
k	int	No	Number of top recommendations to return per item (default: 5)
metadata	pandas.DataFrame	Yes	DataFrame holding metadata for all items (for get_top_k_recommendations)
query_id	str	Yes	ID of the item to get recommendations for (for get_top_k_recommendations)

Outputs

Name	Type	Description
clean_dataframe return	pandas.DataFrame	DataFrame with a new column containing cleaned, concatenated text
tokenize_text return	(TfidfVectorizer, pandas.Series)	The configured vectorizer and tokenized text series
get_tokens return	dict	Dictionary of tokens generated by the TF-IDF vectorizer
get_stop_words return	frozenset	Stop words excluded by the TF-IDF vectorizer
recommend_top_k_items return	pandas.DataFrame	DataFrame with columns: id_col, rec_rank, rec_score, rec_{id_col}
get_top_k_recommendations return	pandas.Styler	Stylized DataFrame with top-k recommendations enriched with metadata

Usage Examples

Basic Usage

from recommenders.models.tfidf.tfidf_utils import TfidfRecommender
import pandas as pd

# Initialize recommender with NLTK tokenization
recommender = TfidfRecommender(id_col="paper_id", tokenization_method="nltk")

# Clean text columns
df_clean = recommender.clean_dataframe(
    df, cols_to_clean=["abstract", "full_text"]
)

# Tokenize and fit
tf, vectors_tokenized = recommender.tokenize_text(df_clean, text_col="cleaned_text")
recommender.fit(tf, vectors_tokenized)

# Get top 10 recommendations for all items
top_k_df = recommender.recommend_top_k_items(df_clean, k=10)

# Get enriched recommendations for a specific item
results = recommender.get_top_k_recommendations(
    metadata=df,
    query_id="my_paper_id",
    cols_to_keep=["title", "authors", "url"]
)