Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness

Abstract

The accuracy of deep learning, i.e., deep neural networks, can be characterized by dividing the total error into three main types: approximation error, optimization error, and generalization error. Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization. Most existing theoretical works for generalization fail to explain the performance of neural networks in practice. To derive a meaningful bound, we study the generalization error of neural networks for classification problems in terms of data distribution and neural network smoothness. We introduce the cover complexity (CC) to measure the difficulty of learning a data set and the inverse of the modulus of continuity to quantify neural network smoothness. A quantitative bound for expected accuracy/error is derived by considering both the CC and neural network smoothness. Although most of the analysis is general and not specific to neural networks, we validate our theoretical assumptions and results numerically for neural networks by several data sets of images. The numerical results confirm that the expected error of trained networks scaled with the square root of the number of classes has a linear relationship with respect to the CC. We also observe a clear consistency between test loss and neural network smoothness during the training process. In addition, we demonstrate empirically that the neural network smoothness decreases when the network size increases whereas the smoothness is insensitive to training dataset size.

References in corpus (10)

• Understanding deep learning requires rethinking generalization
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Generalization in Deep Networks: The Role of Distance from Initialization
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience

Cited by in corpus (3)

• Deep Hamiltonian networks based on symplectic integrators
• Certified machine learning: Rigorous a posteriori error bounds for PDE defined PINNs
• SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems