Principle:Scikit learn Scikit learn Manifold Learning

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Unsupervised Learning, Dimensionality Reduction
Last Updated	2026-02-08 15:00 GMT

Overview

Manifold learning discovers the low-dimensional structure underlying high-dimensional data by preserving local or global geometric properties of the data manifold.

Description

Manifold learning methods are non-linear dimensionality reduction techniques based on the assumption that high-dimensional data often lies on or near a lower-dimensional manifold embedded in the ambient space. Unlike linear methods such as PCA, manifold learning can uncover curved, non-linear structures. These techniques solve the problem of visualizing and understanding the intrinsic geometry of complex datasets where linear projections fail to capture meaningful structure. They sit within unsupervised learning and are primarily used for exploratory data analysis and visualization, though some methods also produce embeddings useful for downstream tasks.

Usage

Use t-SNE for visualizing high-dimensional data in 2D or 3D, particularly when local cluster structure is important -- it excels at preserving neighborhood relationships. Use Isomap when the data lies on a globally smooth manifold and geodesic distances should be preserved. Use Locally Linear Embedding (LLE) when local linearity is a reasonable assumption and computational efficiency matters. Use MDS (Multidimensional Scaling) when you want to preserve pairwise distances in the low-dimensional embedding. Use Spectral Embedding for graph-structured data or when the data's connectivity structure is more meaningful than its Euclidean geometry.

Theoretical Basis

t-SNE (t-distributed Stochastic Neighbor Embedding):

1. Compute pairwise affinities in high-dimensional space using a Gaussian kernel:

   $p_{j|i}=\frac{\exp (-\Vert x_i-x_j{\Vert }^2/2{\sigma }_i^2)}{\sum _{k\ne i}\exp (-\Vert x_i-x_k{\Vert }^2/2{\sigma }_i^2)}$

   Symmetrize: $p_{ij}=(p_{j|i}+p_{i|j})/2n$

2. Define affinities in low-dimensional space using a Student-t distribution with one degree of freedom:

   $q_{ij}=\frac{(1+\Vert y_i-y_j{\Vert }^2{)}^{-1}}{\sum _{k\ne l}(1+\Vert y_k-y_l{\Vert }^2{)}^{-1}}$

3. Minimize the KL divergence: $KL(P\Vert Q)=\sum _{i\ne j}{p}_{ij}\log \frac{p_{ij}}{q_{ij}}$

The heavy-tailed Student-t distribution in the low-dimensional space alleviates the crowding problem that afflicts Gaussian-based methods.

Isomap preserves geodesic distances:

1. Construct a nearest neighbor graph.
2. Compute shortest-path (geodesic) distances between all pairs using Dijkstra's or Floyd-Warshall algorithm.
3. Apply classical MDS to the geodesic distance matrix.

Locally Linear Embedding (LLE):

1. Express each point as a weighted linear combination of its $k$ nearest neighbors by minimizing:

   $\sum _i\Vert x_i-\sum _j{W}_{ij}{x}_j{\Vert }^2$

2. Find the low-dimensional embedding $Y$ that best preserves these weights:

   $\underset{Y}{\min }\sum _i\Vert y_i-\sum _j{W}_{ij}{y}_j{\Vert }^2$

3. This reduces to an eigenvalue problem on the matrix $M=(I-W{)}^T(I-W)$.

Multidimensional Scaling (MDS):

• Classical MDS: Finds an embedding that preserves inner products derived from the distance matrix via double centering. Equivalent to PCA when distances are Euclidean.
• Metric MDS (Stress minimization): Minimizes:

$\text{Stress}=\sqrt{\sum _{i<j}(d_{ij}-\Vert y_i-y_j\Vert {)}^2}$

Spectral Embedding:

1. Construct an affinity (similarity) graph from the data.
2. Compute the graph Laplacian $L=D-W$ (or its normalized variant).
3. Embed using the eigenvectors corresponding to the smallest non-zero eigenvalues of $L$.

Related Pages

• Implementation:Scikit_learn_Scikit_learn_TSNE
• Implementation:Scikit_learn_Scikit_learn_Isomap
• Implementation:Scikit_learn_Scikit_learn_LocallyLinearEmbedding
• Implementation:Scikit_learn_Scikit_learn_MDS
• Implementation:Scikit_learn_Scikit_learn_SpectralEmbedding
• Implementation:Scikit_learn_Scikit_learn_ClassicalMDS