Stochastic Optimization of PCA with Capped MSG

Abstract

We study PCA as a stochastic optimization problem and propose a novel stochastic approximation algorithm which we refer to as "Matrix Stochastic Gradient" (MSG), as well as a practical variant, Capped MSG. We study the method both theoretically and empirically.

References in corpus (1)

• On the Universality of Online Mirror Descent

Cited by in corpus (21)

• Online Learning: A Comprehensive Survey
• Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation
• Fast, Robust and Non-convex Subspace Recovery
• Near-Optimal Stochastic Approximation for Online Principal Component Estimation
• Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation
• Large-Scale Approximate Kernel Canonical Correlation Analysis
• A Stochastic PCA and SVD Algorithm with an Exponential Convergence Rate
• A Linearly Convergent Algorithm for Distributed Principal Component Analysis
• FAST-PCA: A Fast and Exact Algorithm for Distributed Principal Component Analysis
• Rivalry of Two Families of Algorithms for Memory-Restricted Streaming PCA
• Scale Up Nonlinear Component Analysis with Doubly Stochastic Gradients
• Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations
• Exponentially convergent stochastic k-PCA without variance reduction
• Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis
• Convergence of Stochastic Gradient Descent for PCA
• Communication-Efficient Distributed SVD via Local Power Iterations
• ODE-Inspired Analysis for the Biological Version of Oja's Rule in Solving Streaming PCA
• FedPower: Privacy-Preserving Distributed Eigenspace Estimation
• A C++ library for Multimodal Deep Learning
• Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis
• Convergence Rate of Krasulina Estimator