Principle:Scikit learn Scikit learn Text Feature Extraction

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Natural Language Processing, Feature Engineering
Last Updated	2026-02-08 15:00 GMT

Overview

Text feature extraction converts unstructured text documents into structured numerical feature vectors that machine learning algorithms can process.

Description

Raw text cannot be directly consumed by most machine learning algorithms, which require fixed-length numerical input. Text feature extraction methods transform collections of text documents into numerical matrices by representing each document as a vector in a vocabulary-defined feature space. These techniques solve the problem of bridging the gap between natural language and mathematical models. The most common approaches are bag-of-words representations (count-based and TF-IDF weighted), feature hashing for memory-efficient encoding, and dictionary vectorization for pre-structured feature dictionaries. Text feature extraction is the foundational preprocessing step in text classification, clustering, and information retrieval.

Usage

Use CountVectorizer to obtain raw term frequency vectors, suitable as a baseline and when downstream algorithms handle normalization. Use TfidfVectorizer to produce TF-IDF weighted vectors that down-weight common terms and up-weight discriminative ones, typically yielding better performance for classification and retrieval tasks. Use FeatureHasher when the vocabulary is very large or unknown in advance and memory efficiency is critical, accepting the trade-off of losing interpretability. Use DictVectorizer when features arrive as dictionaries of feature-value pairs rather than raw text. Use stop word removal to filter out common, non-informative words that add noise.

Theoretical Basis

Bag of Words represents a document as a vector of term frequencies, ignoring word order:

$x_{j} = count of term j in document$

The vocabulary $V = {t_{1}, t_{2}, \dots, t_{D}}$ defines the feature space of dimension $D$ .

TF-IDF (Term Frequency-Inverse Document Frequency) weights each term by its discriminative power:

$tf-idf (t, d) = tf (t, d) \times idf (t)$

where:

Term Frequency: $tf (t, d) = f_{t, d}$ (raw count or sub-linear: $1 + \log f_{t, d}$ )
Inverse Document Frequency: $idf (t) = \log \frac{1 + n}{1 + df (t)} + 1$

where $n$ is the total number of documents and $df (t)$ is the number of documents containing term $t$ . The "+1" terms provide smoothing.

After TF-IDF weighting, documents are typically L2-normalized:

$\hat{x} = \frac{x}{‖ x ‖_{2}}$

Feature Hashing (the hashing trick) maps features to a fixed-size vector using a hash function:

$x_{j} = \sum_{i : h (t_{i}) = j} ξ (t_{i}) \cdot value (t_{i})$

where $h$ is a hash function mapping terms to indices in ${0, \dots, m - 1}$ and $ξ$ is a sign function that reduces collision effects. This avoids storing a vocabulary dictionary, enabling constant-memory operation.

N-grams extend the bag-of-words model by considering sequences of $n$ consecutive tokens:

Unigrams: individual words
Bigrams: pairs of consecutive words
Character n-grams: sequences of $n$ characters, useful for capturing morphological patterns and handling misspellings

Stop Words are high-frequency words (e.g., "the", "is", "at") that carry little discriminative information. Removing them reduces dimensionality and noise.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment