Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization

Abstract

Dimension reduction (DR) techniques such as t-SNE, UMAP, and TriMAP have demonstrated impressive visualization performance on many real world datasets. One tension that has always faced these methods is the trade-off between preservation of global structure and preservation of local structure: these methods can either handle one or the other, but not both. In this work, our main goal is to understand what aspects of DR methods are important for preserving both local and global structure: it is difficult to design a better method without a true understanding of the choices we make in our algorithms and their empirical impact on the lower-dimensional embeddings they produce. Towards the goal of local structure preservation, we provide several useful design principles for DR loss functions based on our new understanding of the mechanisms behind successful DR methods. Towards the goal of global structure preservation, our analysis illuminates that the choice of which components to preserve is important. We leverage these insights to design a new algorithm for DR, called Pairwise Controlled Manifold Approximation Projection (PaCMAP), which preserves both local and global structure. Our work provides several unexpected insights into what design choices both to make and avoid when constructing DR algorithms.

References in corpus (8)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Efficient Algorithms for t-distributed Stochastic Neighborhood Embedding
• A Survey on Metric Learning for Feature Vectors and Structured Data
• Automatic Selection of t-SNE Perplexity
• TriMap: Large-scale Dimensionality Reduction Using Triplets

Cited by in corpus (9)

• Theoretical Foundations of t-SNE for Visualizing High-Dimensional Clustered Data
• Exploring UMAP in hybrid models of entropy-based and representativeness sampling for active learning in biomedical segmentation
• IAN: Iterated Adaptive Neighborhoods for manifold learning and dimensionality estimation
• A Riemannian Stochastic Representation for Quantifying Model Uncertainties in Molecular Dynamics Simulations
• A Classification-by-Retrieval Framework for Few-Shot Anomaly Detection to Detect API Injection Attacks
• Calibrating dimension reduction hyperparameters in the presence of noise
• Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes
• A Probabilistic Graph Coupling View of Dimension Reduction
• Improving Metric Dimensionality Reduction with Distributed Topology