Transformers for Modeling Physical Systems

Abstract

Transformers are widely used in natural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the natural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature.

22 pages, 14 figures, 3 appendices

References in corpus (6)

• Extended dynamic mode decomposition with dictionary learning: a data-driven adaptive spectral decomposition of the Koopman operator
• Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition
• Augmenting Self-attention with Persistent Memory
• Modern Koopman Theory for Dynamical Systems
• Prediction of laminar vortex shedding over a cylinder using deep learning
• Tensorized Transformer for Dynamical Systems Modeling

Cited by in corpus (20)

• Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems
• PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEs
• Deep Neural Networks with Koopman Operators for Modeling and Control of Autonomous Vehicles
• State estimation with limited sensors -- A deep learning based approach
• AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storage
• Neural Integral Equations
• Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations
• DNN-MG: A Hybrid Neural Network/Finite Element Method with Applications to 3D Simulations of the Navier-Stokes Equations
• Multi-scale Time-stepping of Partial Differential Equations with Transformers
• Towards Complex Dynamic Physics System Simulation with Graph Neural ODEs
• LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations without Simulated Data
• Mapping the X-ray variability of GRS1915+105 with machine learning
• CKNet: A Convolutional Neural Network Based on Koopman Operator for Modeling Latent Dynamics from Pixels
• Transformer models as an efficient replacement for statistical test suites to evaluate the quality of random numbers
• Koopman Learning with Episodic Memory
• TAEN: A Model-Constrained Tikhonov Autoencoder Network for Forward and Inverse Problems
• On Deep-Learning-Based Closures for Algebraic Surrogate Models of Turbulent Flows
• Data-Augmented Predictive Deep Neural Network: Enhancing the extrapolation capabilities of non-intrusive surrogate models
• Spatiotemporal System Forecasting with Irregular Time Steps via Masked Autoencoder
• Orthogonal Transforms in Neural Networks Amount to Effective Regularization