Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data

Abstract

Throughout many fields, practitioners often rely on differential equations to model systems. Yet, for many applications, the theoretical derivation of such equations and/or accurate resolution of their solutions may be intractable. Instead, recently developed methods, including those based on parameter estimation, operator subset selection, and neural networks, allow for the data-driven discovery of both ordinary and partial differential equations (PDEs), on a spectrum of interpretability. The success of these strategies is often contingent upon the correct identification of representative equations from noisy observations of state variables and, as importantly and intertwined with that, the mathematical strategies utilized to enforce those equations. Specifically, the latter has been commonly addressed via unconstrained optimization strategies. Representing the PDE as a neural network, we propose to discover the PDE by solving a constrained optimization problem and using an intermediate state representation similar to a Physics-Informed Neural Network (PINN). The objective function of this constrained optimization problem promotes matching the data, while the constraints require that the PDE is satisfied at several spatial collocation points. We present a penalty method and a widely used trust-region barrier method to solve this constrained optimization problem, and we compare these methods on numerical examples. Our results on the Burgers' and the Korteweg-De Vreis equations demonstrate that the latter constrained method outperforms the penalty method, particularly for higher noise levels or fewer collocation points. For both methods, we solve these discovered neural network PDEs with classical methods, such as finite difference methods, as opposed to PINNs-type methods relying on automatic differentiation. We briefly highlight other small, yet crucial, implementation details.

Minor changes: added references, clarified use of classical solvers over PINNs, minor notation changes. 39 pages, 18 figures

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Fourier Neural Operator for Parametric Partial Differential Equations
• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Evaluating time series forecasting models: An empirical study on performance estimation methods
• Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
• Efficient training of physics-informed neural networks via importance sampling
• SINDy-PI: A Robust Algorithm for Parallel Implicit Sparse Identification of Nonlinear Dynamics
• Weak SINDy For Partial Differential Equations
• PySINDy: A comprehensive Python package for robust sparse system identification
• Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
• Data-Driven Deep Learning of Partial Differential Equations in Modal Space
• SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs
• Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusion
• A practical PINN framework for multi-scale problems with multi-magnitude loss terms
• Learning partial differential equations for biological transport models from noisy spatiotemporal data
• Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data
• Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
• Symbolic Regression is NP-hard
• Open Problems in Applied Deep Learning
• Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving PDEs with sharp solutions
• Data-driven identification of stable differential operators using constrained regression
• Linear Stability Analysis of Physics-Informed Random Projection Neural Networks for ODEs

Cited by in corpus (1)

• CTSR: Cartesian tensor-based sparse regression for data-driven discovery of high-dimensional invariant governing equations