Why gradient clipping accelerates training: A theoretical justification for adaptivity

Abstract

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, \emph{gradient clipping} and \emph{normalized gradient}, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

References in corpus (13)

• On the difficulty of training Recurrent Neural Networks
• Convolutional Sequence to Sequence Learning
• On the Convergence of Adam and Beyond
• Recent Trends in Deep Learning Based Natural Language Processing
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• A Universal Catalyst for First-Order Optimization
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• The Power of Normalization: Faster Evasion of Saddle Points
• Stochastic Nested Variance Reduction for Nonconvex Optimization
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• Linear Convergence of Variance-Reduced Stochastic Gradient without Strong Convexity

Cited by in corpus (36)

• High-Performance Large-Scale Image Recognition Without Normalization
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective
• Recent Advances of Differential Privacy in Centralized Deep Learning: A Systematic Survey
• SGD-Net: Efficient Model-Based Deep Learning with Theoretical Guarantees
• Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• A hybrid learning method for system identification and optimal control
• Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness
• Zero-touch Continuous Network Slicing Control via Scalable Actor-Critic Learning
• Privacy-Preserving News Recommendation Model Learning
• Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descent
• GOAT: GPU Outsourcing of Deep Learning Training With Asynchronous Probabilistic Integrity Verification Inside Trusted Execution Environment
• Leveraging Non-uniformity in First-order Non-convex Optimization
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Stochastic Training is Not Necessary for Generalization
• Learning compositional functions via multiplicative weight updates
• Non-convex Distributionally Robust Optimization: Non-asymptotic Analysis
• Monte-Carlo Graph Search for AlphaZero
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
• Activated Gradients for Deep Neural Networks
• On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
• On Proximal Policy Optimization's Heavy-tailed Gradients
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes
• Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
• Multilevel Monte Carlo Variational Inference
• Improving Differentially Private SGD via Randomly Sparsified Gradients
• Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
• Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
• Understanding the Role of Adversarial Regularization in Supervised Learning