Multi-scale Time-stepping of Partial Differential Equations with Transformers
arXiv:2311.02225 · doi:10.1016/j.cma.2024.116983
Abstract
Developing fast surrogates for Partial Differential Equations (PDEs) will accelerate design and optimization in almost all scientific and engineering applications. Neural networks have been receiving ever-increasing attention and demonstrated remarkable success in computational modeling of PDEs, however; their prediction accuracy is not at the level of full deployment. In this work, we utilize the transformer architecture, the backbone of numerous state-of-the-art AI models, to learn the dynamics of physical systems as the mixing of spatial patterns learned by a convolutional autoencoder. Moreover, we incorporate the idea of multi-scale hierarchical time-stepping to increase the prediction speed and decrease accumulated error over time. Our model achieves similar or better results in predicting the time-evolution of Navier-Stokes equations compared to the powerful Fourier Neural Operator (FNO) and two transformer-based neural operators OFormer and Galerkin Transformer.
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
- Machine Learning for Fluid Mechanics
- Longformer: The Long-Document Transformer
- Fourier Neural Operator for Parametric Partial Differential Equations
- Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
- PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
- Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks
- Learning to Simulate Complex Physics with Graph Networks
- Nonlinear mode decomposition with convolutional neural networks for fluid dynamics
- Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders
- Convolutional neural network based hierarchical autoencoder for nonlinear mode decomposition of fluid field data
- Solving parametric PDE problems with artificial neural networks
- Transformers for Modeling Physical Systems
- Multiwavelet-based Operator Learning for Differential Equations
- Choose a Transformer: Fourier or Galerkin
- Learning Operators with Coupled Attention
- Transformer for Partial Differential Equations' Operator Learning
- GNOT: A General Neural Operator Transformer for Operator Learning
- Towards Multi-spatiotemporal-scale Generalized PDE Modeling
- Scalable Transformer for PDE Surrogate Modeling
- Transformer Meets Boundary Value Inverse Problems
- Generalizable data-driven turbulence closure modeling on unstructured grids with differentiable physics