In-Context Operator Learning with Data Prompts for Differential Equation Problems

Abstract

This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON), to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Existing methods are limited to using a neural network to approximate a specific equation solution or a specific operator, requiring retraining when switching to a new problem with different equations. By training a single neural network as an operator learner, we can not only get rid of retraining (even fine-tuning) the neural network for new problems, but also leverage the commonalities shared across operators so that only a few demos in the prompt are needed when learning a new operator. Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems, including forward and inverse problems of ordinary differential equations (ODEs), partial differential equations (PDEs), and mean-field control (MFC) problems, and also show that it can generalize its learning capability to operators beyond the training distribution.

arXiv commentThe second and third authors contributed equally. This is an outdated preprint. Please refer to the updated version published in PNAS: www.pnas.org/doi/10.1073/pnas.2310142120 See code in https://github.com/LiuYangMage/in-context-operator-networks

References in corpus (25)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Fourier Neural Operator for Parametric Partial Differential Equations
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Weak Adversarial Networks for High-dimensional Partial Differential Equations
• Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios
• Transfer learning based multi-fidelity physics informed deep neural network
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems
• Solving parametric PDE problems with artificial neural networks
• Deep transfer operator learning for partial differential equations under conditional shift
• Reliable extrapolation of deep neural operators informed by physics or sparse observations
• Analysis of three dimensional potential problems in non-homogeneous media with physics-informed deep collocation method using material transfer learning and sensitivity analysis
• Learning Functional Priors and Posteriors from Data and Physics
• Multi-fidelity prediction of fluid flow and temperature field based on transfer learning using Fourier Neural Operator
• Learning Operators with Coupled Attention
• Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains
• In-Context Operator Learning with Data Prompts for Differential Equation Problems
• Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow
• Meta-Auto-Decoder for Solving Parametric Partial Differential Equations
• SVD-PINNs: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decomposition
• One-Shot Transfer Learning of Physics-Informed Neural Networks

Cited by in corpus (6)

• In-Context Operator Learning with Data Prompts for Differential Equation Problems
• Combining physics-based and data-driven models: advancing the frontiers of research with Scientific Machine Learning
• Synergistic Learning with Multi-Task DeepONet for Efficient PDE Problem Solving
• Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods
• Magnetic Hysteresis Modeling with Neural Operators
• CompNO: A Novel Foundation Model approach for solving Partial Differential Equations