Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning

Abstract

A supervised learning problem is to find a function in a hypothesis function space given values on isolated data points. Inspired by the frequency principle in neural networks, we propose a Fourier-domain variational formulation for supervised learning problem. This formulation circumvents the difficulty of imposing the constraints of given values on isolated data points in continuum modelling. Under a necessary and sufficient condition within our unified framework, we establish the well-posedness of the Fourier-domain variational problem, by showing a critical exponent depending on the data dimension. In practice, a neural network can be a convenient way to implement our formulation, which automatically satisfies the well-posedness condition.

18 pages, 5 figures

References in corpus (13)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• On the Spectral Bias of Neural Networks
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Theory of the Frequency Principle for General Deep Neural Networks
• Multi-scale Deep Neural Network (MscaleDNN) Methods for Oscillatory Stokes Flows in Complex Domains
• Multi-scale Convolutional Neural Networks for Inverse Problems
• Asymptotic behavior of ${\ell }_p$-based Laplacian regularization in semi-supervised learning
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• A type of generalization error induced by initialization in deep neural networks