Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform

Abstract

Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning. This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expensive. To address this challenge, we present two main contributions: (1) A new efficient retraction map based on an iterative Cayley transform for optimization updates, and (2) An implicit vector transport mechanism based on the combination of a projection of the momentum and the Cayley transform on the Stiefel manifold. We specify two new optimization algorithms: Cayley SGD with momentum, and Cayley ADAM on the Stiefel manifold. Convergence of Cayley SGD is theoretically analyzed. Our experiments for CNN training demonstrate that both algorithms: (a) Use less running time per iteration relative to existing approaches that enforce orthonormality of CNN parameters; and (b) Achieve faster convergence rates than the baseline SGD and ADAM algorithms without compromising the performance of the CNN. Cayley SGD and Cayley ADAM are also shown to reduce the training time for optimizing the unitary transition matrices in RNNs.

arXiv commentICLR 2020 conference paper

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Wide Residual Networks
• Full-Capacity Unitary Recurrent Neural Networks
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation

Cited by in corpus (13)

• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
• Continual Learning in Low-rank Orthogonal Subspaces
• Differentiable Programming of Isometric Tensor Networks
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• What if Neural Networks had SVDs?
• Orthogonal Over-Parameterized Training
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• Learning Bijective Feature Maps for Linear ICA
• Generative Kernel Spectral Clustering
• Generalized One-Class Learning Using Pairs of Complementary Classifiers
• CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices