Implicit bias with Ritz-Galerkin method in understanding deep learning for solving PDEs

Abstract

This paper aims at studying the difference between Ritz-Galerkin (R-G) method and deep neural network (DNN) method in solving partial differential equations (PDEs) to better understand deep learning. To this end, we consider solving a particular Poisson problem, where the information of the right-hand side of the equation f is only available at n sample points, that is, f is known at finite sample points. Through both theoretical and numerical studies, we show that solution of the R-G method converges to a piecewise linear function for the one dimensional (1D) problem or functions of lower regularity for high dimensional problems. With the same setting, DNNs however learn a relative smooth solution regardless of the dimension, this is, DNNs implicitly bias towards functions with more low-frequency components among all functions that can fit the equation at available data points. This bias is explained by the recent study of frequency principle (Xu et al., (2019) [17] and Zhang et al., (2019) [11, 19]). In addition to the similarity between the traditional numerical methods and DNNs in the approximation perspective, our work shows that the implicit bias in the learning process, which is different from traditional numerical methods, could help better understand the characteristics of DNNs.

arXiv comment16 pages, 34 figures

References in corpus (11)

• Solving high-dimensional partial differential equations using deep learning
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions
• Deep least-squares methods: an unsupervised learning-based numerical method for solving elliptic PDEs
• A mesh-free method for interface problems using the deep learning approach
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• PhaseDNN - A Parallel Phase Shift Deep Neural Network for Adaptive Wideband Learning
• Approximation Rates for Neural Networks with General Activation Functions

Cited by in corpus (1)

• SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs