Linear Dimensionality Reduction: Survey, Insights, and Generalizations

Abstract

Linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data, due to their simple geometric interpretations and typically attractive computational properties. These methods capture many data features of interest, such as covariance, dynamical structure, correlation between data sets, input-output relationships, and margin between data classes. Methods have been developed with a variety of names and motivations in many fields, and perhaps as a result the connections between all these methods have not been highlighted. Here we survey methods from this disparate literature as optimization programs over matrix manifolds. We discuss principal component analysis, factor analysis, linear multidimensional scaling, Fisher's linear discriminant analysis, canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, distance metric learning, and more. This optimization framework gives insight to some rarely discussed shortcomings of well-known methods, such as the suboptimality of certain eigenvector solutions. Modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver, which accepts as input data and an objective to be optimized, and returns, as output, an optimal low-dimensional projection of the data. This simple optimization framework further allows straightforward generalizations and novel variants of classical methods, which we demonstrate here by creating an orthogonal-projection canonical correlations analysis. More broadly, this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox, objective-agnostic numerical technology.

42 pages, 5 figures, 1 table

References in corpus (3)

• Manopt, a Matlab toolbox for optimization on manifolds
• Generalized power method for sparse principal component analysis
• Kernel dimension reduction in regression

Cited by in corpus (66)

• Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
• Optimization Problems for Machine Learning: A Survey
• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• Sparse Principal Component Analysis via Variable Projection
• Randomized Matrix Decompositions using R
• Abnormal Client Behavior Detection in Federated Learning
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data
• Geometric Mean Metric Learning
• Data-Driven Reduced-Order Modeling of Spatiotemporal Chaos with Neural Ordinary Differential Equations
• Uncertainty-Aware Principal Component Analysis
• Interactive Dimensionality Reduction for Comparative Analysis
• A supervised learning approach involving active subspaces for an efficient genetic algorithm in high-dimensional optimization problems
• Transforming Feature Space to Interpret Machine Learning Models
• Correlated Components Analysis - Extracting Reliable Dimensions in Multivariate Data
• SLISEMAP: Supervised dimensionality reduction through local explanations
• Go with the FLOW: Visualizing spatiotemporal dynamics in optical widefield calcium imaging
• Nonlinear Dimensionality Reduction for Data Visualization: An Unsupervised Fuzzy Rule-based Approach
• Machine Learning Techniques and Applications For Ground-based Image Analysis
• A Tutorial on Distance Metric Learning: Mathematical Foundations, Algorithms, Experimental Analysis, Prospects and Challenges (with Appendices on Mathematical Background and Detailed Algorithms Explanation)
• Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis
• ParaDime: A Framework for Parametric Dimensionality Reduction
• Learning in High-Dimensional Multimedia Data: The State of the Art
• Bayesian Methods for Joint Exoplanet Transit Detection and Systematic Noise Characterization
• Machine Learning with World Knowledge: The Position and Survey
• How to reduce dimension with PCA and random projections?
• Multipartite Pooling for Deep Convolutional Neural Networks
• Outlier Aware Network Embedding for Attributed Networks
• Factorized Discriminant Analysis for Genetic Signatures of Neuronal Phenotypes
• A Self-consistent-field Iteration for Orthogonal Canonical Correlation Analysis
• Quantitative Evaluation of Time-Dependent Multidimensional Projection Techniques
• A Theory of the NEPv Approach for Optimization On the Stiefel Manifold
• Single-Server Private Linear Transformation: The Individual Privacy Case
• Fast algorithms for robust principal component analysis with an upper bound on the rank
• Online landmark replacement for out-of-sample dimensionality reduction methods
• A Differentially Private Probabilistic Framework for Modeling the Variability Across Federated Datasets of Heterogeneous Multi-View Observations
• Too many secants: a hierarchical approach to secant-based dimensionality reduction on large data sets
• Random Projection in Deep Neural Networks
• On Feature Reduction using Deep Learning for Trend Prediction in Finance
• Simultaneously Learning Neighborship and Projection Matrix for Supervised Dimensionality Reduction
• Grassmann Iterative Linear Discriminant Analysis with Proxy Matrix Optimization
• Dimensionality Reduction on Grassmannian via Riemannian Optimization: A Generalized Perspective
• Asymmetric Random Projections
• Bayesian neural networks and dimensionality reduction
• Ordinal Constrained Binary Code Learning for Nearest Neighbor Search
• Meta-learning One-class Classifiers with Eigenvalue Solvers for Supervised Anomaly Detection
• Trace Ratio Optimization with an Application to Multi-view Learning
• HyperNP: Interactive Visual Exploration of Multidimensional Projection Hyperparameters
• Using Slisemap to interpret physical data
• A Visual Analytics Framework for Reviewing Multivariate Time-Series Data with Dimensionality Reduction
• Supervised Discriminative Sparse PCA with Adaptive Neighbors for Dimensionality Reduction
• Spatial Filtering for Brain Computer Interfaces: A Comparison between the Common Spatial Pattern and Its Variant
• Joint Learning of Discriminative Low-dimensional Image Representations Based on Dictionary Learning and Two-layer Orthogonal Projections
• An Overview of Computational Approaches for Interpretation Analysis
• Joint Projection and Dictionary Learning using Low-rank Regularization and Graph Constraints
• Grassmannian Discriminant Maps (GDM) for Manifold Dimensionality Reduction with Application to Image Set Classification
• Multiple Manifolds Metric Learning with Application to Image Set Classification
• The Effectiveness of Johnson-Lindenstrauss Transform for High Dimensional Optimization With Adversarial Outliers, and the Recovery
• Use of High Dimensional Modeling for automatic variables selection: the best path algorithm
• Locality Sensitive Hash Aggregated Nonlinear Neighbourhood Matrix Factorization for Online Sparse Big Data Analysis
• Shape-Preserving Dimensionality Reduction : An Algorithm and Measures of Topological Equivalence
• Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey
• A Local Similarity-Preserving Framework for Nonlinear Dimensionality Reduction with Neural Networks
• Subsampling Bias and The Best-Discrepancy Systematic Cross Validation
• Multipartite Ranking-Selection of Low-Dimensional Instances by Supervised Projection to High-Dimensional Space
• Manifold Optimization Assisted Gaussian Variational Approximation