Over-parametrized deep neural networks do not generalize well

Abstract

Recently it was shown in several papers that backpropagation is able to find the global minimum of the empirical risk on the training data using over-parametrized deep neural networks. In this paper a similar result is shown for deep neural networks with the sigmoidal squasher activation function in a regression setting, and a lower bound is presented which proves that these networks do not generalize well on a new data in the sense that they do not achieve the optimal minimax rate of convergence for estimation of smooth regression functions.

References in corpus (6)

• Deep Learning in Neural Networks: An Overview
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Does data interpolation contradict statistical optimality?
• Adding One Neuron Can Eliminate All Bad Local Minima
• On the rate of convergence of fully connected very deep neural network regression estimates
• On the rate of convergence of a neural network regression estimate learned by gradient descent

Cited by in corpus (4)

• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• Statistical theory for image classification using deep convolutional neural networks with cross-entropy loss under the hierarchical max-pooling model
• Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
• Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping