Three Factors Influencing Minima in SGD
arXiv:1711.04623
Abstract
We investigate the dynamical and convergent properties of stochastic gradient descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the relation between learning rate, batch size and the properties of the final minima, such as width or generalization, remains an open question. In order to tackle this problem we investigate the previously proposed approximation of SGD by a stochastic differential equation (SDE). We theoretically argue that three factors - learning rate, batch size and gradient covariance - influence the minima found by SGD. In particular we find that the ratio of learning rate to batch size is a key determinant of SGD dynamics and of the width of the final minima, and that higher values of the ratio lead to wider minima and often better generalization. We confirm these findings experimentally. Further, we include experiments which show that learning rate schedules can be replaced with batch size schedules and that the ratio of learning rate to batch size is an important factor influencing the memorization process.
arXiv commentFirst two authors contributed equally. Short version accepted into ICLR workshop. Accepted to Artificial Neural Networks and Machine Learning, ICANN 2018
References in corpus (10)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Understanding deep learning requires rethinking generalization
- One weird trick for parallelizing convolutional neural networks
- Opening the Black Box of Deep Neural Networks via Information
- Stochastic Gradient Hamiltonian Monte Carlo
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Qualitatively characterizing neural network optimization problems
- Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
- (Non-) asymptotic properties of Stochastic Gradient Langevin Dynamics
Cited by in corpus (128)
- A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- Quantum Natural Gradient
- Revisiting Small Batch Training for Deep Neural Networks
- Training Tips for the Transformer Model
- Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
- Don't Use Large Mini-Batches, Use Local SGD
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Dark Experience for General Continual Learning: a Strong, Simple Baseline
- Learning explanations that are hard to vary
- A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
- A Modern Take on the Bias-Variance Tradeoff in Neural Networks
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
- Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
- Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
- Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
- Fluctuation-dissipation relations for stochastic gradient descent
- On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
- Understanding Batch Normalization
- On the Origin of Implicit Regularization in Stochastic Gradient Descent
- The Heavy-Tail Phenomenon in SGD
- Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
- Explicit Regularisation in Gaussian Noise Injections
- Federated Learning with Buffered Asynchronous Aggregation
- Mean-field inference methods for neural networks
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima
- The Implicit Regularization of Stochastic Gradient Flow for Least Squares
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks
- The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- The Implicit and Explicit Regularization Effects of Dropout
- A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
- Mathematical Models of Overparameterized Neural Networks
- Non-Gaussianity of Stochastic Gradient Noise
- Recent advances in deep learning theory
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
- Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
- Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
- Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
- On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
- SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
- On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
- Large Scale Structure of Neural Network Loss Landscapes
- Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
- Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
- Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
- On Learning Rates and Schrödinger Operators
- Towards Understanding Generalization in Gradient-Based Meta-Learning
- CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
- F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
- First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
- RankNEAT: Outperforming Stochastic Gradient Search in Preference Learning Tasks
- An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
- A consensus-based global optimization method for high dimensional machine learning problems
- Positively Scale-Invariant Flatness of ReLU Neural Networks
- Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
- How noise affects the Hessian spectrum in overparameterized neural networks
- Large-Scale Deep Learning Optimizations: A Comprehensive Survey
- Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
- Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
- Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms
- Deep Learning is Singular, and That's Good
- On the interplay between noise and curvature and its effect on optimization and generalization
- Is Support Set Diversity Necessary for Meta-Learning?
- Needles in Haystacks: On Classifying Tiny Objects in Large Images
- Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
- Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting
- Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
- Regularizing Neural Networks via Adversarial Model Perturbation
- On Accelerating Distributed Convex Optimizations
- Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
- Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
- A Loss Curvature Perspective on Training Instability in Deep Learning
- Unifying Regularisation Methods for Continual Learning
- AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
- On generalization bounds for deep networks based on loss surface implicit regularization
- Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
- Retrosynthesis with Attention-Based NMT Model and Chemical Analysis of the "Wrong" Predictions
- Parameter Re-Initialization through Cyclical Batch Size Schedules
- The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
- Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
- SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
- Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
- Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
- Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function
- Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
- What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
- Improved generalization by noise enhancement
- Hyperplane Arrangements of Trained ConvNets Are Biased
- Fast, Better Training Trick -- Random Gradient
- Improving the convergence of SGD through adaptive batch sizes
- Robustness, Privacy, and Generalization of Adversarial Training
- Critical Learning Periods in Federated Learning
- Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
- Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise
- Smoothness Analysis of Adversarial Training
- Understanding Short-Range Memory Effects in Deep Neural Networks
- On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
- Trap of Feature Diversity in the Learning of MLPs
- The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
- Subaging in underparametrized Deep Neural Networks
- What training reveals about neural network complexity
- On the Bias-Variance Tradeoff: Textbooks Need an Update
- Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
- Unique Properties of Flat Minima in Deep Networks
- Enhance Diffusion to Improve Robust Generalization
- Generalisation in fully-connected neural networks for time series forecasting
- Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
- SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
- Pushing the boundaries of parallel Deep Learning -- A practical approach
- Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
- How do SGD hyperparameters in natural training affect adversarial robustness?
- Backtracking gradient descent method for general functions, with applications to Deep Learning
- Inherent Noise in Gradient Based Methods
- Efficient Classification of Very Large Images with Tiny Objects
- Analytic expressions for the output evolution of a deep neural network
- A Sharp Convergence Rate for the Asynchronous Stochastic Gradient Descent
- On Large Batch Training and Sharp Minima: A Fokker-Planck Perspective
- What can linear interpolation of neural network loss landscapes tell us?
- LRTuner: A Learning Rate Tuner for Deep Neural Networks
- Adaptive norms for deep learning with regularized Newton methods
- Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
- Deep Learning Theory through the Lens of Statistical Physics: A Topical Review
- A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit
- BN-invariant sharpness regularizes the training model to better generalization
- Continuous-time Models for Stochastic Optimization Algorithms
- Relative Flatness and Generalization