Quasi-hyperbolic momentum and Adam for deep learning

Abstract

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

arXiv commentPublished as a conference paper at ICLR 2019. This version corrects one typological error in the published text

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• An overview of gradient descent optimization algorithms
• Convolutional Sequence to Sequence Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Addressing Function Approximation Error in Actor-Critic Methods
• On the Convergence of Adam and Beyond
• EMNIST: an extension of MNIST to handwritten letters
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• YellowFin and the Art of Momentum Tuning
• Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning

Cited by in corpus (16)

• On Empirical Comparisons of Optimizers for Deep Learning
• AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
• Review: Deep Learning in Electron Microscopy
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• On the adequacy of untuned warmup for adaptive optimization
• NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning
• On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Attention augmented differentiable forest for tabular data
• Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
• An Adaptive Remote Stochastic Gradient Method for Training Neural Networks
• Flexible numerical optimization with ensmallen
• Variance Reduction in Deep Learning: More Momentum is All You Need