SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations

Abstract

The least squares method with deep neural networks as function parametrization has been applied to solve certain high-dimensional partial differential equations (PDEs) successfully; however, its convergence is slow and might not be guaranteed even within a simple class of PDEs. To improve the convergence of the network-based least squares model, we introduce a novel self-paced learning framework, SelectNet, which quantifies the difficulty of training samples, treats samples equally in the early stage of training, and slowly explores more challenging samples, e.g., samples with larger residual errors, mimicking the human cognitive process for more efficient learning. In particular, a selection network and the PDE solution network are trained simultaneously; the selection network adaptively weighting the training samples of the solution network achieving the goal of self-paced learning. Numerical examples indicate that the proposed SelectNet model outperforms existing models on the convergence speed and the convergence robustness, especially for low-regularity solutions.

arXiv comment23 pages, 27 figures; revised

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Weak Adversarial Networks for High-dimensional Partial Differential Equations
• Some Further Results for the Stationary Points and Dynamics of Supercooled Liquids
• Why Deep Neural Networks for Function Approximation?
• Optimal approximation of continuous functions by very deep ReLU networks
• D3M: A deep domain decomposition method for partial differential equations
• Deep splitting method for parabolic PDEs
• Neural Network Approximation: Three Hidden Layers Are Enough
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Nonlinear Approximation via Compositions
• Approximation capability of two hidden layer feedforward neural networks with fixed weights
• Computing Committor Functions for the Study of Rare Events Using Deep Learning
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Overcoming the curse of dimensionality in the approximative pricing of financial derivatives with default risks
• A Priori Estimates of the Population Risk for Residual Networks
• Integrating Machine Learning with Physics-Based Modeling
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• SelectNet: Learning to Sample from the Wild for Imbalanced Data Training
• Enhanced Expressive Power and Fast Training of Neural Networks by Random Projections

Cited by in corpus (14)

• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• GAS: A Gaussian Mixture Distribution-Based Adaptive Sampling Method for PINNs
• Neural Networks Enforcing Physical Symmetries in Nonlinear Dynamical Lattices: The Case Example of the Ablowitz-Ladik Model
• Structure Probing Neural Network Deflation
• Gradient and Uncertainty Enhanced Sequential Sampling for Global Fit
• Reproducing Activation Function for Deep Learning
• On the Representation of Solutions to Elliptic PDEs in Barron Spaces
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• Learn bifurcations of nonlinear parametric systems via equation-driven neural networks
• A learning scheme by sparse grids and Picard approximations for semilinear parabolic PDEs
• Stationary Density Estimation of Itô Diffusions Using Deep Learning