Principle:Rapidsai Cuml Dimensionality Reduction Algorithm Selection

Knowledge Sources	UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction Visualizing Data using t-SNE Principal Component Analysis
Domains	Machine_Learning, Dimensionality_Reduction
Last Updated	2026-02-08 00:00 GMT

Overview

Dimensionality reduction algorithm selection is the process of choosing between linear methods (PCA, SVD) and nonlinear methods (UMAP, t-SNE) based on the structure of the data, the goals of the analysis, and computational constraints.

Description

High-dimensional datasets often contain redundant or correlated features that can be compressed into a lower-dimensional representation without significant loss of information. The choice of dimensionality reduction algorithm depends on several factors:

Linear methods such as PCA (Principal Component Analysis) assume that the most important variation in the data can be captured by linear combinations of the original features. PCA performs an eigendecomposition of the covariance matrix (or equivalently, a Singular Value Decomposition of the data matrix) to find orthogonal directions of maximum variance. These methods are fast, deterministic, and invertible, making them suitable for data compression, noise removal, and preprocessing pipelines.

Nonlinear methods capture complex manifold structure that linear methods miss:

• UMAP (Uniform Manifold Approximation and Projection) is grounded in topological data analysis. It constructs a fuzzy simplicial set representation of the high-dimensional data's local neighborhood structure, then optimizes a low-dimensional layout that preserves this topological representation. UMAP balances local and global structure preservation and supports out-of-sample transforms.

• t-SNE (t-Distributed Stochastic Neighbor Embedding) converts pairwise distances into conditional probabilities in both high-dimensional and low-dimensional spaces, then minimizes the Kullback-Leibler (KL) divergence between these distributions. It excels at revealing cluster structure for visualization but is transductive (cannot project new points) and does not preserve global distances.

Usage

Use this principle when deciding which dimensionality reduction algorithm to apply:

• Choose PCA when you need a fast, deterministic, invertible linear projection, when explained variance is a meaningful metric, or when the data has approximately linear structure. Common use cases include data compression, denoising, and feature extraction for downstream models.
• Choose UMAP when you need to preserve both local and global manifold structure, when you require out-of-sample projection, or when working with large datasets where performance matters. UMAP is suitable for both visualization and as a general-purpose embedding.
• Choose t-SNE when the primary goal is 2D/3D visualization of cluster structure and local neighborhoods. t-SNE is particularly effective at revealing well-separated clusters but should not be used when out-of-sample projection is needed.

Theoretical Basis

PCA: Eigendecomposition of the Covariance Matrix

Given a data matrix $X\in {\mathbb{ℝ}}^{n\times d}$ (mean-centered), PCA finds the directions of maximum variance by solving:

$\mathrm{\Sigma }=\frac{1}{n-1}{X}^{T}X$

$\mathrm{\Sigma }{v}_i={\lambda }_i{v}_i$

where $\mathrm{\Sigma }$ is the covariance matrix, $v_i$ are the principal components (eigenvectors), and ${\lambda }_i$ are the explained variances (eigenvalues). The top $k$ components define the projection $Z=X{V}_k$.

UMAP: Fuzzy Simplicial Sets

UMAP constructs a weighted k-neighbor graph in high-dimensional space, where edge weights represent the probability that two points are connected. The algorithm:

1. Computes k-nearest neighbors and local distances ${\rho }_i$ and bandwidth ${\sigma }_i$.
2. Constructs a fuzzy simplicial set by combining local metric spaces using fuzzy set union.
3. Optimizes a low-dimensional embedding by minimizing the cross-entropy between the high-dimensional and low-dimensional fuzzy simplicial sets:

$C=\sum _{e\in E}[w_h(e)\log \frac{w_h(e)}{w_l(e)}+(1-w_h(e))\log \frac{1-w_h(e)}{1-w_l(e)}]$

t-SNE: KL Divergence Minimization

t-SNE converts pairwise Euclidean distances into joint probabilities $p_{ij}$ in high-dimensional space (Gaussian kernel) and $q_{ij}$ in low-dimensional space (Student's t-distribution with one degree of freedom):

$p_{j|i}=\frac{\exp (-\Vert x_i-x_j{\Vert }^2/2{\sigma }_i^2)}{\sum _{k\ne i}\exp (-\Vert x_i-x_k{\Vert }^2/2{\sigma }_i^2)}$

$q_{ij}=\frac{(1+\Vert y_i-y_j{\Vert }^2{)}^{-1}}{\sum _{k\ne l}(1+\Vert y_k-y_l{\Vert }^2{)}^{-1}}$

The embedding is found by minimizing the KL divergence:

$KL(P\Vert Q)=\sum _{i\ne j}{p}_{ij}\log \frac{p_{ij}}{q_{ij}}$

Algorithm Selection Decision Pseudocode

function select_algorithm(data, goal):
    if goal == "compression" or goal == "preprocessing":
        return PCA  # Fast, invertible, deterministic
    elif goal == "visualization":
        if need_out_of_sample_transform:
            return UMAP
        elif dataset_size > 50000:
            return UMAP  # Scales better than t-SNE
        else:
            return TSNE  # Best cluster separation for visualization
    elif goal == "manifold_learning":
        if need_transform_on_new_data:
            return UMAP
        else:
            return UMAP or TSNE  # Both capture nonlinear structure

Related Pages

Implemented By

• Implementation:Rapidsai_Cuml_PCA_UMAP_TSNE_Configuration