Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network

Abstract

Neural operators have recently become popular tools for designing solution maps between function spaces in the form of neural networks. Differently from classical scientific machine learning approaches that learn parameters of a known partial differential equation (PDE) for a single instance of the input parameters at a fixed resolution, neural operators approximate the solution map of a family of PDEs. Despite their success, the uses of neural operators are so far restricted to relatively shallow neural networks and confined to learning hidden governing laws. In this work, we propose a novel nonlocal neural operator, which we refer to as nonlocal kernel network (NKN), that is resolution independent, characterized by deep neural networks, and capable of handling a variety of tasks such as learning governing equations and classifying images. Our NKN stems from the interpretation of the neural network as a discrete nonlocal diffusion reaction equation that, in the limit of infinite layers, is equivalent to a parabolic nonlocal equation, whose stability is analyzed via nonlocal vector calculus. The resemblance with integral forms of neural operators allows NKNs to capture long-range dependencies in the feature space, while the continuous treatment of node-to-node interactions makes NKNs resolution independent. The resemblance with neural ODEs, reinterpreted in a nonlocal sense, and the stable network dynamics between layers allow for generalization of NKN's optimal parameters from shallow to deep networks. This fact enables the use of shallow-to-deep initialization techniques. Our tests show that NKNs outperform baseline methods in both learning governing equations and image classification tasks and generalize well to different resolutions and depths.

References in corpus (16)

• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Fourier Neural Operator for Parametric Partial Differential Equations
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks
• Solving ill-posed inverse problems using iterative deep neural networks
• Deep Equilibrium Models
• Numerical methods for nonlocal and fractional models
• Solving parametric PDE problems with artificial neural networks
• EikoNet: Solving the Eikonal equation with Deep Neural Networks
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
• Graph Neural Ordinary Differential Equations
• Implicit Graph Neural Networks
• Multiscale Deep Equilibrium Models
• Continuous Graph Neural Networks
• Generalized Joint Probability Density Function Formulation inTurbulent Combustion using DeepONet

Cited by in corpus (7)

• Reliable extrapolation of deep neural operators informed by physics or sparse observations
• Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems
• Long-term predictions of turbulence by implicit U-Net enhanced Fourier neural operator
• Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems
• Adaptive physics-informed neural operator for coarse-grained non-equilibrium flows
• Harnessing Scale and Physics: A Multi-Graph Neural Operator Framework for PDEs on Arbitrary Geometries
• Uncertainty quantification and stability of neural operators for prediction of three-dimensional turbulence