Transformers for Modeling Physical Systems

Abstract

Transformers are widely used in natural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the natural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature.

arXiv comment22 pages, 14 figures, 3 appendices

References in corpus (33)

• Efficient Estimation of Word Representations in Vector Space
• Neural Machine Translation by Jointly Learning to Align and Translate
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Self-Attention Generative Adversarial Networks
• Recent Advances in Physical Reservoir Computing: A Review
• Deep learning for universal linear embeddings of nonlinear dynamics
• Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control
• Complex Patterns in a Simple System
• Effective Approaches to Attention-based Neural Machine Translation
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Neural Ordinary Differential Equations
• Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
• Extended dynamic mode decomposition with dictionary learning: a data-driven adaptive spectral decomposition of the Koopman operator
• Learning to Simulate Complex Physics with Graph Networks
• A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems
• Deep convolutional encoder-decoder networks for uncertainty quantification of dynamic multiphase flow in heterogeneous media
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders
• A new prediction method of unsteady wake flow by the hybrid deep neural network
• Multi-level Convolutional Autoencoder Networks for Parametric Prediction of Spatio-temporal Dynamics
• Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM
• The Neural Particle Method -- An Updated Lagrangian Physics Informed Neural Network for Computational Fluid Dynamics
• Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition
• Augmented Neural ODEs
• Learning Compositional Koopman Operators for Model-Based Control
• Augmenting Self-attention with Persistent Memory
• Structured Bayesian Gaussian process latent variable model: applications to data-driven dimensionality reduction and high-dimensional inversion
• Non-intrusive model reduction of large-scale, nonlinear dynamical systems using deep learning
• Multi-fidelity Generative Deep Learning Turbulent Flows
• Modern Koopman Theory for Dynamical Systems
• Prediction of laminar vortex shedding over a cylinder using deep learning
• Tensorized Transformer for Dynamical Systems Modeling

Cited by in corpus (20)

• Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems
• PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEs
• Deep Neural Networks with Koopman Operators for Modeling and Control of Autonomous Vehicles
• State estimation with limited sensors -- A deep learning based approach
• AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storage
• Neural Integral Equations
• Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations
• DNN-MG: A Hybrid Neural Network/Finite Element Method with Applications to 3D Simulations of the Navier-Stokes Equations
• Multi-scale Time-stepping of Partial Differential Equations with Transformers
• Towards Complex Dynamic Physics System Simulation with Graph Neural ODEs
• LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations without Simulated Data
• Mapping the X-ray variability of GRS1915+105 with machine learning
• CKNet: A Convolutional Neural Network Based on Koopman Operator for Modeling Latent Dynamics from Pixels
• Transformer models as an efficient replacement for statistical test suites to evaluate the quality of random numbers
• Koopman Learning with Episodic Memory
• TAEN: A Model-Constrained Tikhonov Autoencoder Network for Forward and Inverse Problems
• On Deep-Learning-Based Closures for Algebraic Surrogate Models of Turbulent Flows
• Data-Augmented Predictive Deep Neural Network: Enhancing the extrapolation capabilities of non-intrusive surrogate models
• Spatiotemporal System Forecasting with Irregular Time Steps via Masked Autoencoder
• Orthogonal Transforms in Neural Networks Amount to Effective Regularization