LaProp: Separating Momentum and Adaptivity in Adam

Abstract

We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• An overview of gradient descent optimization algorithms
• On the Convergence of Adam and Beyond
• On the Variance of the Adaptive Learning Rate and Beyond
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Lookahead Optimizer: k steps forward, 1 step back
• Averaging Weights Leads to Wider Optima and Better Generalization
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Optimization for deep learning: theory and algorithms
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• A general system of differential equations to model first order adaptive algorithms

Cited by in corpus (9)

• t-Soft Update of Target Network for Deep Reinforcement Learning
• Deep Reinforcement Learning Control of Quantum Cartpoles
• Neural Networks Fail to Learn Periodic Functions and How to Fix It
• Learning Not to Learn in the Presence of Noisy Labels
• Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning
• Convergent and Efficient Deep Q Network Algorithm
• Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures
• On the Distributional Properties of Adaptive Gradients
• Volumization as a Natural Generalization of Weight Decay