A Surprising Linear Relationship Predicts Test Performance in Deep Networks

Abstract

Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalization may improve practical applications of deep networks. In this paper we show that with cross-entropy loss it is surprisingly simple to induce significantly different generalization performances for two networks that have the same architecture, the same meta parameters and the same training error: one can either pretrain the networks with different levels of "corrupted" data or simply initialize the networks with weights of different Gaussian standard deviations. A corollary of recent theoretical results on overfitting shows that these effects are due to an intrinsic problem of measuring test performance with a cross-entropy/exponential-type loss, which can be decomposed into two components both minimized by SGD -- one of which is not related to expected classification performance. However, if we factor out this component of the loss, a linear relationship emerges between training and test losses. Under this transformation, classical generalization bounds are surprisingly tight: the empirical/training loss is very close to the expected/test loss. Furthermore, the empirical relation between classification error and normalized cross-entropy loss seem to be approximately monotonic

References in corpus (1)

• Theory II: Landscape of the Empirical Risk in Deep Learning

Cited by in corpus (11)

• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Implicit Regularization in Over-parameterized Neural Networks
• An Effective Label Noise Model for DNN Text Classification
• In Search of Robust Measures of Generalization
• Why flatness does and does not correlate with generalization for deep neural networks
• Distribution of Classification Margins: Are All Data Equal?
• Rethinking Breiman's Dilemma in Neural Networks: Phase Transitions of Margin Dynamics
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks