Accelerated Optimization With Orthogonality Constraints

Abstract

We develop a generalization of Nesterov's accelerated gradient descent method which is designed to deal with orthogonality constraints. To demonstrate the effectiveness of our method, we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number, and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold. Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large, ill-conditioned problems.

arXiv comment17 pages, 4 figures

References in corpus (3)

• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• Compressed Modes for Variational Problems in Mathematics and Physics
• Towards Riemannian Accelerated Gradient Methods

Cited by in corpus (6)

• Machine Learning Universal Bosonic Functionals
• From Nesterov's Estimate Sequence to Riemannian Acceleration
• Generalized Left-Localized Cayley Parametrization for Optimization with Orthogonality Constraints
• Solving Optimization Problems over the Stiefel Manifold by Smooth Exact Penalty Function
• Learning Bijective Feature Maps for Linear ICA
• On some limitations of probabilistic models for dimension-reduction: Illustration in the case of probabilistic formulations of partial least squares