Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
arXiv:2004.13954
Abstract
Over-parameterized deep neural networks (DNNs) with sufficient capacity to memorize random noise can achieve excellent generalization performance, challenging the bias-variance trade-off in classical learning theory. Recent studies claimed that DNNs first learn simple patterns and then memorize noise; some other works showed a phenomenon that DNNs have a spectral bias to learn target functions from low to high frequencies during training. However, we show that the monotonicity of the learning bias does not always hold: under the experimental setup of deep double descent, the high-frequency components of DNNs diminish in the late stage of training, leading to the second descent of the test error. Besides, we find that the spectrum of DNNs can be applied to indicating the second descent of the test error, even though it is calculated from the training set only.
arXiv commentIJCAI-21. Int'l Joint. Conf. on Artificial Intelligence (IJCAI), Montreal, Canada, August 2021
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- A Closer Look at Memorization in Deep Networks
- Exponential expressivity in deep neural networks through transient chaos
- Scaling description of generalization with number of parameters in deep learning
- On the number of response regions of deep feed forward networks with piece-wise linear activations
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
- Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
- SGD on Neural Networks Learns Functions of Increasing Complexity
- Identity Crisis: Memorization and Generalization under Extreme Overparameterization
- Empirical Studies on the Properties of Linear Regions in Deep Neural Networks