Transformer for Partial Differential Equations' Operator Learning

Abstract

Data-driven learning of partial differential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models that are built upon problem-specific inductive biases. An example is a convolutional or a graph neural network that exploits the local grid structure where functions' values are sampled. The attention mechanism, on the other hand, provides a flexible way to implicitly exploit the patterns within inputs, and furthermore, relationship between arbitrary query locations and inputs. In this work, we present an attention-based framework for data-driven operator learning, which we term Operator Transformer (OFormer). Our framework is built upon self-attention, cross-attention, and a set of point-wise multilayer perceptrons (MLPs), and thus it makes few assumptions on the sampling pattern of the input function or query locations. We show that the proposed framework is competitive on standard benchmark problems and can flexibly be adapted to randomly sampled input.

Cited by in corpus (10)

• Variational operator learning: A unified paradigm marrying training neural operators and solving partial differential equations
• Physics-informed MeshGraphNets (PI-MGNs): Neural finite element solvers for non-stationary and nonlinear simulations on arbitrary meshes
• Multi-scale Time-stepping of Partial Differential Equations with Transformers
• Operator Learning with Neural Fields: Tackling PDEs on General Geometries
• Quantum DeepONet: Neural operators accelerated by quantum computing
• Deciphering and integrating invariants for neural operator learning with various physical mechanisms
• Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates
• A Deep Learning approach for parametrized and time dependent Partial Differential Equations using Dimensionality Reduction and Neural ODEs
• Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems
• High Throughput Training of Deep Surrogates from Large Ensemble Runs