Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs

Abstract

The optimization with orthogonality has been shown useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, both computational efficiency and stability are important. However, existing methods utilizing Riemannian optimization or hard constraints can only ensure stability while those using soft constraints can only improve efficiency. In this paper, we propose a novel method, named Feedback Gradient Descent (FGD), to our knowledge, the first work showing high efficiency and stability simultaneously. FGD induces orthogonality based on the simple yet indispensable Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold. In particular, inspired by a numerical integration method on manifolds called Feedback Integrators, we propose to instantiate it on the tangent bundle of the Stiefel manifold for the first time. In the extensive image classification experiments, FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability.

References in corpus (30)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On the difficulty of training Recurrent Neural Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Neural Photo Editing with Introspective Adversarial Networks
• Unitary Evolution Recurrent Neural Networks
• On orthogonality and learning recurrent networks with long term dependencies
• On the Matrix Inversion Approximation Based on Neumann Series in Massive MIMO Systems
• Full-Capacity Unitary Recurrent Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Accelerating SGD with momentum for over-parameterized learning
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• Trivializations for Gradient-Based Optimization on Manifolds
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• On Controller Design for Systems on Manifolds in Euclidean Space
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• Deep Isometric Learning for Visual Recognition
• Projection Based Weight Normalization for Deep Neural Networks
• Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Training Group Orthogonal Neural Networks with Privileged Information
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Adaptive and Momentum Methods on Manifolds Through Trivializations
• Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
• Tracking Controller Design for Satellite Attitude Under Unknown Constant Disturbance Using Stable Embedding