Theory IIIb: Generalization in Deep Networks

Abstract

A main puzzle of deep neural networks (DNNs) revolves around the apparent absence of "overfitting", defined in this paper as follows: the expected error does not get worse when increasing the number of neurons or of iterations of gradient descent. This is surprising because of the large capacity demonstrated by DNNs to fit randomly labeled data and the absence of explicit regularization. Recent results by Srebro et al. provide a satisfying solution of the puzzle for linear networks used in binary classification. They prove that minimization of loss functions such as the logistic, the cross-entropy and the exp-loss yields asymptotic, "slow" convergence to the maximum margin solution for linearly separable datasets, independently of the initial conditions. Here we prove a similar result for nonlinear multilayer DNNs near zero minima of the empirical loss. The result holds for exponential-type losses but not for the square loss. In particular, we prove that the weight matrix at each layer of a deep network converges to a minimum norm solution up to a scale factor (in the separable case). Our analysis of the dynamical system corresponding to gradient descent of a multilayer network suggests a simple criterion for ranking the generalization performance of different zero minimizers of the empirical loss.

arXiv comment38 pages, 7 figures

References in corpus (2)

• Robust Large Margin Deep Neural Networks
• Theory II: Landscape of the Empirical Risk in Deep Learning

Cited by in corpus (7)

• Are All Layers Created Equal?
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Deep Neural Networks for Choice Analysis: A Statistical Learning Theory Perspective
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks
• Think Global, Act Local: Relating DNN generalisation and node-level SNR
• Layer Dynamics of Linearised Neural Nets