AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Abstract

Convolutional neural networks (CNNs) are trained using stochastic gradient descent (SGD)-based optimizers. Recently, the adaptive moment estimation (Adam) optimizer has become very popular due to its adaptive momentum, which tackles the dying gradient problem of SGD. Nevertheless, existing optimizers are still unable to exploit the optimization curvature information efficiently. This paper proposes a new AngularGrad optimizer that considers the behavior of the direction/angle of consecutive gradients. This is the first attempt in the literature to exploit the gradient angular information apart from its magnitude. The proposed AngularGrad generates a score to control the step size based on the gradient angular information of previous iterations. Thus, the optimization steps become smoother as a more accurate step size of immediate past gradients is captured through the angular information. Two variants of AngularGrad are developed based on the use of Tangent or Cosine functions for computing the gradient angular information. Theoretically, AngularGrad exhibits the same regret bound as Adam for convergence purposes. Nevertheless, extensive experiments conducted on benchmark data sets against state-of-the-art methods reveal a superior performance of AngularGrad. The source code will be made publicly available at: https://github.com/mhaut/AngularGrad.

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Decoupled Weight Decay Regularization
• On the difficulty of training Recurrent Neural Networks
• Recent Trends in Deep Learning Based Natural Language Processing
• Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Online Learning Rate Adaptation with Hypergradient Descent
• Rethinking the Hyperparameters for Fine-tuning
• Dual Path Networks

Cited by in corpus (2)

• Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers