Normalized Direction-preserving Adam

Abstract

Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this work, we identify the reasons that Adam generalizes worse than SGD, and develop a variant of Adam to eliminate the generalization gap. The proposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and step size for updating weight vectors, leading to significantly improved generalization performance. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also hope to shed light on why certain optimization algorithms generalize better than others.

References in corpus (9)

• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• A Closer Look at Memorization in Deep Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Beyond Convexity: Stochastic Quasi-Convex Optimization

Cited by in corpus (9)

• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• A Survey of Optimization Methods from a Machine Learning Perspective
• Jasper: An End-to-End Convolutional Neural Acoustic Model
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• A general system of differential equations to model first order adaptive algorithms
• A Thousand Words are Worth More Than One Recording: NLP Based Speaker Change Point Detection
• DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
• ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization