RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm

Abstract

We propose a novel and efficient training method for RNNs by iteratively seeking a local minima on the loss surface within a small region, and leverage this directional vector for the update, in an outer-loop. We propose to utilize the Frank-Wolfe (FW) algorithm in this context. Although, FW implicitly involves normalized gradients, which can lead to a slow convergence rate, we develop a novel RNN training method that, surprisingly, even with the additional cost, the overall training cost is empirically observed to be lower than back-propagation. Our method leads to a new Frank-Wolfe method, that is in essence an SGD algorithm with a restart scheme. We prove that under certain conditions our algorithm has a sublinear convergence rate of $O(1/\epsilon )$ for $\epsilon$ error. We then conduct empirical experiments on several benchmark datasets including those that exhibit long-term dependencies, and show significant performance improvement. We also experiment with deep RNN architectures and show efficient training performance. Finally, we demonstrate that our training method is robust to noisy data.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• On the difficulty of training Recurrent Neural Networks
• Trust Region Policy Optimization
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• How to Construct Deep Recurrent Neural Networks
• Latent ODEs for Irregularly-Sampled Time Series
• On orthogonality and learning recurrent networks with long term dependencies
• On the Convergence Rate of Training Recurrent Neural Networks
• Full-Capacity Unitary Recurrent Neural Networks
• Stable Recurrent Models
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Improving performance of recurrent neural network with relu nonlinearity
• Unbiasing Truncated Backpropagation Through Time
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Learning Long Term Dependencies via Fourier Recurrent Units
• Recurrent Neural Networks in the Eye of Differential Equations
• SGD Converges to Global Minimum in Deep Learning via Star-convex Path
• Kronecker Recurrent Units
• RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?
• Time-Delay Momentum: A Regularization Perspective on the Convergence and Generalization of Stochastic Momentum for Deep Learning