Activated Gradients for Deep Neural Networks

Abstract

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

References in corpus (8)

• An overview of gradient descent optimization algorithms
• Gaussian Error Linear Units (GELUs)
• Searching for Activation Functions
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning