Principle:DistrictDataLabs Yellowbrick Manifold Embedding

Knowledge Sources	Yellowbrick Docs Yellowbrick van der Maaten & Hinton, 2008 Tenenbaum et al., 2000 Roweis & Saul, 2000
Domains	Machine_Learning, Feature_Analysis, Visualization
Last Updated	2026-02-08 00:00 GMT

Overview

Manifold embedding is a family of non-linear dimensionality reduction techniques that project high-dimensional data into a low-dimensional space while preserving the intrinsic geometric structure of the data manifold.

Description

Unlike linear methods such as PCA, manifold learning algorithms assume the data lies on or near a low-dimensional non-linear surface (manifold) embedded in the high-dimensional feature space. These algorithms attempt to "unfold" this manifold into two or three dimensions for visualization while preserving local or global geometric relationships such as distances, neighborhoods, or geodesic paths.

Different manifold algorithms capture different aspects of the data geometry:

• t-SNE (t-distributed Stochastic Neighbor Embedding) preserves local neighborhood structure and excels at revealing clusters, but does not preserve global distances.
• Isomap approximates geodesic distances along the manifold using shortest paths in a neighborhood graph, preserving global structure.
• Locally Linear Embedding (LLE) and its variants (LTSA, Hessian, Modified) reconstruct each point as a weighted combination of its neighbors, preserving local linear structure.
• MDS (Multi-Dimensional Scaling) preserves pairwise distances between all points.
• Spectral Embedding computes a low-dimensional representation using the eigenvectors of the graph Laplacian.

Each algorithm has distinct trade-offs in computational cost, memory usage, and the type of structure it preserves, making algorithm selection an important part of the analysis.

Usage

Manifold embedding is used to:

• Visualize non-linear structure that PCA cannot capture.
• Discover clusters in data that are not linearly separable.
• Explore latent structure in high-dimensional feature spaces.
• Compare multiple algorithms to understand different aspects of data geometry.
• Validate feature engineering by checking whether transformed features reveal expected patterns.

Theoretical Basis

t-SNE

t-SNE models pairwise similarities as conditional probabilities. In the high-dimensional space, the similarity of point $x_j$ to $x_i$ is:

$p_{j|i}=\frac{\exp (-\Vert x_i-x_j{\Vert }^2/2{\sigma }_i^2)}{\sum _{k\ne i}\exp (-\Vert x_i-x_k{\Vert }^2/2{\sigma }_i^2)}$

In the low-dimensional space, a Student-t distribution with one degree of freedom is used:

$q_{ij}=\frac{(1+\Vert y_i-y_j{\Vert }^2{)}^{-1}}{\sum _{k\ne l}(1+\Vert y_k-y_l{\Vert }^2{)}^{-1}}$

The embedding is found by minimizing the Kullback-Leibler divergence between the two distributions.

Isomap

Isomap extends MDS by replacing Euclidean distances with geodesic distances estimated by shortest paths in a k-nearest-neighbor graph:

1. Construct a k-nearest-neighbor graph.
2. Compute shortest-path distances $d_G(i,j)$ between all pairs using Dijkstra's algorithm.
3. Apply classical MDS to the geodesic distance matrix.

Locally Linear Embedding

LLE operates in three steps:

1. Find the k nearest neighbors for each point.
2. Compute the reconstruction weights $W_{ij}$ that best reconstruct each point from its neighbors by minimizing $\sum _i\Vert x_i-\sum _j{W}_{ij}{x}_j{\Vert }^2$.
3. Find the low-dimensional embedding by minimizing $\sum _i\Vert y_i-\sum _j{W}_{ij}{y}_j{\Vert }^2$ with fixed weights.

Related Pages

Implemented By

• Implementation:DistrictDataLabs_Yellowbrick_Manifold_Visualizer