Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates

Abstract

Neural surrogates for partial differential equations (PDEs) have become popular due to their potential to quickly simulate physics. With a few exceptions, neural surrogates generally treat the forward evolution of time-dependent PDEs as a black box by directly predicting the next state. While this is a natural and easy framework for applying neural surrogates, it can be an over-simplified and rigid framework for predicting physics. In this work, we evaluate an alternate framework in which neural solvers predict the temporal derivative and an ODE integrator forwards the solution in time, which has little overhead and is broadly applicable across model architectures and PDEs. We find that by simply changing the training target and introducing numerical integration during inference, neural surrogates can gain accuracy and stability in finely-discretized regimes. Predicting temporal derivatives also allows models to not be constrained to a specific temporal discretization, allowing for flexible time-stepping during inference or training on higher-resolution PDE data. Lastly, we investigate why this framework can be beneficial and in what situations does it work well.

22 pages, 7 figures. For code see http://github.com/anthonyzhou-1/temporal_pdes

References in corpus (45)

• Denoising Diffusion Probabilistic Models
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows
• Visualizing the Loss Landscape of Neural Nets
• Learning data driven discretizations for partial differential equations
• FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
• Hamiltonian Neural Networks
• Learning to Simulate Complex Physics with Graph Networks
• Machine learning force fields and coarse-grained variables in molecular dynamics: application to materials and biological systems
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Solving inverse-PDE problems with physics-aware neural networks
• Graph Neural Networks Accelerated Molecular Dynamics
• GenCast: Diffusion-based ensemble forecasting for medium-range weather
• Message Passing Neural PDE Solvers
• Choose a Transformer: Fourier or Galerkin
• PDEBENCH: An Extensive Benchmark for Scientific Machine Learning
• Transformer for Partial Differential Equations' Operator Learning
• U-NO: U-shaped Neural Operators
• Geometry-Informed Neural Operator for Large-Scale 3D PDEs
• Towards Multi-spatiotemporal-scale Generalized PDE Modeling
• Learned Coarse Models for Efficient Turbulence Simulation
• PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers
• Scalable Transformer for PDE Surrogate Modeling
• DNN-MG: A Hybrid Neural Network/Finite Element Method with Applications to 3D Simulations of the Navier-Stokes Equations
• Self-supervised Learning with Physics-aware Neural Networks I: Galaxy Model Fitting
• Multi-scale Time-stepping of Partial Differential Equations with Transformers
• Learning to correct spectral methods for simulating turbulent flows
• Transolver: A Fast Transformer Solver for PDEs on General Geometries
• Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs
• Learning Dissipative Dynamics in Chaotic Systems
• Lie Point Symmetry Data Augmentation for Neural PDE Solvers
• Poseidon: Efficient Foundation Models for PDEs
• From Zero to Turbulence: Generative Modeling for 3D Flow Simulation
• The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
• Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modeling
• Strategies for Pretraining Neural Operators
• Masked Autoencoders are PDE Learners
• A Neural PDE Solver with Temporal Stencil Modeling
• Text2PDE: Latent Diffusion Models for Accessible Physics Simulation
• PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systems
• CaFA: Global Weather Forecasting with Factorized Attention on Sphere