Structure Probing Neural Network Deflation
arXiv:2007.03609 · doi:10.1016/j.jcp.2021.110231
Abstract
Deep learning is a powerful tool for solving nonlinear differential equations, but usually, only the solution corresponding to the flattest local minimizer can be found due to the implicit regularization of stochastic gradient descent. This paper proposes a network-based structure probing deflation method to make deep learning capable of identifying multiple solutions that are ubiquitous and important in nonlinear physical models. First, we introduce deflation operators built with known solutions to make known solutions no longer local minimizers of the optimization energy landscape. Second, to facilitate the convergence to the desired local minimizer, a structure probing technique is proposed to obtain an initial guess close to the desired local minimizer. Together with neural network structures carefully designed in this paper, the new regularized optimization can converge to new solutions efficiently. Due to the mesh-free nature of deep learning, the proposed method is capable of solving high-dimensional problems on complicated domains with multiple solutions, while existing methods focus on merely one or two-dimensional regular domains and are more expensive in operation counts. Numerical experiments also demonstrate that the proposed method could find more solutions than exiting methods.
References in corpus (31)
- Adam: A Method for Stochastic Optimization
- Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
- DGM: A deep learning algorithm for solving partial differential equations
- Solving high-dimensional partial differential equations using deep learning
- Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data
- On the Convergence of Adam and Beyond
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Complex Patterns in a Simple System
- A Convergence Theory for Deep Learning via Over-Parameterization
- A unified deep artificial neural network approach to partial differential equations in complex geometries
- Weak Adversarial Networks for High-dimensional Partial Differential Equations
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Deep Network Approximation for Smooth Functions
- Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
- Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions
- When and why PINNs fail to train: A neural tangent kernel perspective
- Geometry of Optimization and Implicit Regularization in Deep Learning
- Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems
- SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
- Regularity theory and high order numerical methods for the (1d)-Fractional Laplacian
- Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
- Eigenvalues of the fractional Laplace operator in the unit ball
- Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
- Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
- Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
- Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent
- Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implications
- A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems
- PhaseDNN - A Parallel Phase Shift Deep Neural Network for Adaptive Wideband Learning
- A Deflation Technique for Detecting Multiple Liquid Crystal Equilibrium States
- Distinct solutions of finite-dimensional complementarity problems