Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks

Abstract

It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from low to high frequencies. Inspired by the F-Principle, we propose an effective model of linear F-Principle (LFP) dynamics which accurately predicts the learning results of two-layer ReLU neural networks (NNs) of large widths. This LFP dynamics is rationalized by a linearized mean field residual dynamics of NNs. Importantly, the long-time limit solution of this LFP dynamics is equivalent to the solution of a constrained optimization problem explicitly minimizing an FP-norm, in which higher frequencies of feasible solutions are more heavily penalized. Using this optimization formulation, an a priori estimate of the generalization error bound is provided, revealing that a higher FP-norm of the target function increases the generalization error. Overall, by explicitizing the implicit bias of the F-Principle as an explicit penalty for two-layer NNs, our work makes a step towards a quantitative understanding of the learning and generalization of general DNNs.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Understanding deep learning requires rethinking generalization
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A Closer Look at Memorization in Deep Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Spectrally-normalized margin bounds for neural networks
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Failures of Gradient-Based Deep Learning
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Training behavior of deep neural network in frequency domain
• Understanding training and generalization in deep learning by Fourier analysis
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application

Cited by in corpus (14)

• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Training behavior of deep neural network in frequency domain
• A Fine-Grained Spectral Perspective on Neural Networks
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• Structure Probing Neural Network Deflation
• Deep frequency principle towards understanding why deeper learning is faster
• The Slow Deterioration of the Generalization Error of the Random Feature Model
• Implicit bias with Ritz-Galerkin method in understanding deep learning for solving PDEs
• Data-informed Deep Optimization
• Exploring The Effect of High-frequency Components in GANs Training
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning