Exploring Generalization in Deep Learning

Abstract

With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.

19 pages, 8 figures

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Cited by in corpus (2)

• Theory of Deep Learning III: explaining the non-overfitting puzzle
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data