On the interplay between noise and curvature and its effect on optimization and generalization
arXiv:1906.07774
Abstract
The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the variance of the gradients. While most previous works focus on one or the other of these properties, we explore how their interaction affects optimization speed. Further, as the ultimate goal is good generalization performance, we clarify how both curvature and noise are relevant to properly estimate the generalization gap. Realizing that the limitations of some existing works stems from a confusion between these matrices, we also clarify the distinction between the Fisher matrix, the Hessian, and the covariance matrix of the gradients.
arXiv commentAccepted to AISTATS 2020
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Understanding deep learning requires rethinking generalization
- New insights and perspectives on the natural gradient method
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Three Factors Influencing Minima in SGD
- Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
- Sharp Minima Can Generalize For Deep Nets
- Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
- Implicit Regularization in Deep Learning
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- From Averaging to Acceleration, There is Only a Step-size
- Fluctuation-dissipation relations for stochastic gradient descent
- A Scale Invariant Flatness Measure for Deep Network Minima
- Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions
- Real numbers, data science and chaos: How to fit any dataset with a single parameter
Cited by in corpus (4)
- Deep Learning is Singular, and That's Good
- Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
- Scalable and Practical Natural Gradient for Large-Scale Deep Learning
- L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation