RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee
arXiv:2003.03601
Abstract
We investigate online nonlinear regression with continually running recurrent neural network networks (RNNs), i.e., RNN-based online learning. For RNN-based online learning, we introduce an efficient first-order training algorithm that theoretically guarantees to converge to the optimum network parameters. Our algorithm is truly online such that it does not make any assumption on the learning environment to guarantee convergence. Through numerical simulations, we verify our theoretical results and illustrate significant performance improvements achieved by our algorithm with respect to the state-of-the-art RNN training methods.
arXiv commentThis paper was an early draft of the presented results. We have written and published another paper (arXiv:2005.08948) where we have improved the material in this paper. The published paper covers most of the material presented in this paper as well. Therefore, we remove this paper from Arxiv and kindly refer the interested readers to arXiv:2005.08948
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- LSTM: A Search Space Odyssey
- An overview of gradient descent optimization algorithms
- A Critical Review of Recurrent Neural Networks for Sequence Learning
- Gradient Descent Learns Linear Dynamical Systems
- On the Convergence Rate of Training Recurrent Neural Networks
- Stable Recurrent Models
- Regularizing RNNs by Stabilizing Activations
- Stochastic Gradient Descent Learns State Equations with Nonlinear Activations
- A Local Regret in Nonconvex Online Learning