Tensorized Transformer for Dynamical Systems Modeling
arXiv:2006.03445
Abstract
The identification of nonlinear dynamics from observations is essential for the alignment of the theoretical ideas and experimental data. The last, in turn, is often corrupted by the side effects and noise of different natures, so probabilistic approaches could give a more general picture of the process. At the same time, high-dimensional probabilities modeling is a challenging and data-intensive task. In this paper, we establish a parallel between the dynamical systems modeling and language modeling tasks. We propose a transformer-based model that incorporates geometrical properties of the data and provide an iterative training algorithm allowing the fine-grid approximation of the conditional probabilities of high-dimensional dynamical systems.
References in corpus (15)
- Sequence to Sequence Learning with Neural Networks
- Language Models are Few-Shot Learners
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Deep learning for universal linear embeddings of nonlinear dynamics
- On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
- Learning Video Representations using Contrastive Bidirectional Transformer
- Long-time predictive modeling of nonlinear dynamical systems using neural networks
- Long-term Forecasting using Higher Order Tensor RNNs
- Probabilistic Simulation of Quantum Circuits with the Transformer
- Tensorized Embedding Layers for Efficient Model Compression
- Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
- A comparative study of physics-informed neural network models for learning unknown dynamics and constitutive relations
- Turbulence forecasting via Neural ODE
- Self-Attentive Hawkes Processes
- Deep Representation Learning for Dynamical Systems Modeling