Aggregated Momentum: Stability Through Passive Damping

Abstract

Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient $\beta$. Large $\beta$ values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different $\beta$ parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive $\beta$ values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence.

11 primary pages, 11 supplementary pages, 12 figures total

References in corpus (7)

• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Training Deep Nets with Sublinear Memory Cost
• Regularizing and Optimizing LSTM Language Models
• A Variational Perspective on Accelerated Methods in Optimization
• The Reversible Residual Network: Backpropagation Without Storing Activations
• A Lyapunov Analysis of Momentum Methods in Optimization

Cited by in corpus (1)

• Review: Deep Learning in Electron Microscopy