Implementation:Scikit learn Scikit learn TfidfVectorizer

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Natural Language Processing, Feature Extraction
Last Updated	2026-02-08 15:00 GMT

Overview

Concrete tool for converting collections of raw text documents to TF-IDF feature matrices provided by scikit-learn.

Description

TfidfVectorizer converts a collection of raw documents to a matrix of TF-IDF (term frequency-inverse document frequency) features. It is equivalent to CountVectorizer followed by TfidfTransformer. The module also includes CountVectorizer for simple term-count vectorization, TfidfTransformer for normalizing count matrices, and HashingVectorizer for memory-efficient text vectorization. These are the primary tools for text feature extraction in scikit-learn.

Usage

Use TfidfVectorizer when you need to convert text documents into numerical feature vectors for machine learning models. TF-IDF weighting is effective for text classification, clustering, and information retrieval tasks because it down-weights common terms and highlights discriminative terms.

Code Reference

Source Location

• Repository: scikit-learn
• File: sklearn/feature_extraction/text.py

Signature

class TfidfVectorizer(CountVectorizer):
    def __init__(
        self,
        *,
        input="content",
        encoding="utf-8",
        decode_error="strict",
        strip_accents=None,
        lowercase=True,
        preprocessor=None,
        tokenizer=None,
        analyzer="word",
        stop_words=None,
        token_pattern=r"(?u)\b\w\w+\b",
        ngram_range=(1, 1),
        max_df=1.0,
        min_df=1,
        max_features=None,
        vocabulary=None,
        binary=False,
        dtype=np.float64,
        norm="l2",
        use_idf=True,
        smooth_idf=True,
        sublinear_tf=False,
    ):

Import

from sklearn.feature_extraction.text import TfidfVectorizer

I/O Contract

Inputs

Name	Type	Required	Description
input	str	No	Source of input: 'filename', 'file', or 'content'. Default is 'content'.
encoding	str	No	Encoding used to decode bytes. Default is 'utf-8'.
stop_words	str or list	No	Stop words to remove. If 'english', uses built-in list. Default is None.
ngram_range	tuple (min_n, max_n)	No	Range of n-values for n-grams to extract. Default is (1, 1).
max_df	float or int	No	Ignore terms with document frequency above this threshold. Default is 1.0.
min_df	float or int	No	Ignore terms with document frequency below this threshold. Default is 1.
max_features	int	No	Build a vocabulary using only top max_features by term frequency. Default is None.
norm	str	No	Normalization method: 'l1', 'l2', or None. Default is 'l2'.
use_idf	bool	No	Enable inverse-document-frequency reweighting. Default is True.
smooth_idf	bool	No	Smooth IDF weights by adding one to document frequencies. Default is True.
sublinear_tf	bool	No	Apply sublinear TF scaling (replace tf with 1 + log(tf)). Default is False.

Outputs

Name	Type	Description
X_transformed	sparse matrix of shape (n_samples, n_features)	TF-IDF weighted document-term matrix.
vocabulary_	dict	A mapping of terms to feature indices.
idf_	ndarray of shape (n_features,)	Inverse document frequency vector.

Usage Examples

Basic Usage

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "This is the first document.",
    "This document is the second document.",
    "And this is the third one.",
    "Is this the first document?",
]

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())
print(X.shape)
# (4, 9) - 4 documents, 9 unique terms

Related Pages

• Principle:Scikit_learn_Scikit_learn_Text_Feature_Extraction