SVGD: A Virtual Gradients Descent Method for Stochastic Optimization
arXiv:1907.04021
Abstract
Inspired by dynamic programming, we propose Stochastic Virtual Gradient Descent (SVGD) algorithm where the Virtual Gradient is defined by computational graph and automatic differentiation. The method is computationally efficient and has little memory requirements. We also analyze the theoretical convergence properties and implementation of the algorithm. Experimental results on multiple datasets and network models show that SVGD has advantages over other stochastic optimization methods.
arXiv comment12 pages, 12 figures, conference papers
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Training Very Deep Networks
- Adding Gradient Noise Improves Learning for Very Deep Networks
- Neural GPUs Learn Algorithms
- A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
- Neural Programmer: Inducing Latent Programs with Gradient Descent
- A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics