Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
arXiv:1910.00359
Abstract
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role; (4) find that rank does not correlate with generalization or robustness in a practical setting.
arXiv comment18 pages, 6 figures. First two authors contributed equally. Published as a conference paper at ICLR 2020
References in corpus (25)
- Deep Residual Learning for Image Recognition
- Wide Residual Networks
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- The Loss Surfaces of Multilayer Networks
- Certified Defenses against Adversarial Examples
- Deep Neural Networks as Gaussian Processes
- Xception: Deep Learning with Depthwise Separable Convolutions
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- How Does Batch Normalization Help Optimization?
- L2 Regularization versus Batch and Weight Normalization
- Qualitatively characterizing neural network optimization problems
- The Singular Values of Convolutional Layers
- Fixup Initialization: Residual Learning Without Normalization
- Gradient descent aligns the layers of deep linear networks
- On the loss landscape of a class of deep neural networks with no bad local valleys
- Three Mechanisms of Weight Decay Regularization
- Understanding the Loss Surface of Neural Networks for Binary Classification
- Local minima in training of neural networks
- A Kernel Perspective for Regularizing Deep Neural Networks
- Small nonlinearities in activation functions create bad local minima in neural networks
- Understanding Generalization through Visualizations
- Deep Learning without Poor Local Minima
- Piecewise linear activations substantially shape the loss surfaces of neural networks
- Minimum weight norm models do not always generalize well for over-parameterized problems