Training behavior of deep neural network in frequency domain

Abstract

Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.

arXiv commentTo appear in 2019 26th-International conference of neural information processing (ICONIP)

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Understanding deep learning requires rethinking generalization
• Opening the Black Box of Deep Neural Networks via Information
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• A Closer Look at Memorization in Deep Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Generalization Properties and Implicit Regularization for Multiple Passes SGM
• Understanding training and generalization in deep learning by Fourier analysis
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• A type of generalization error induced by initialization in deep neural networks
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
• Meta-learners' learning dynamics are unlike learners'
• PhaseDNN - A Parallel Phase Shift Deep Neural Network for Adaptive Wideband Learning
• Understanding Deep Learning Generalization by Maximum Entropy
• Nonlinear Collaborative Scheme for Deep Neural Networks

Cited by in corpus (13)

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Understanding training and generalization in deep learning by Fourier analysis
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• The Spectral Bias of the Deep Image Prior
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
• One-Shot Transfer Learning of Physics-Informed Neural Networks
• A deep learning-based ODE solver for chemical kinetics
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• Nonlinear Collaborative Scheme for Deep Neural Networks
• Diagnosing Convolutional Neural Networks using their Spectral Response
• Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?