16 papers
Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
Zhaiming Shen, Alexander Cloninger, Rongjie Lai +1
Transformers have emerged as a dominant architecture in modern machine learning, with in-context learning (ICL) enabling models to adapt to new tasks by conditioning on prompts, wi…
Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
Zhongjie Shi, Rongjie Lai, Alexander Cloninger +1
Transformers achieve remarkable performance by jointly learning broad families of tasks during pretraining and adapting to unseen tasks from only a short prompt. Yet a rigorous mat…
Mode Coverage in Normalizing Flow Boltzmann Generators via Log-Ratio Variation
Qi Feng, Rongjie Lai, Di Qi +1
Normalizing flow Boltzmann generators retain a tractable pushforward density, but training with forward KL depends on target samples that may be biased or omit modes. As a result,…
Self-supervised In-context Operator Learning for Stochastic Mean-Field Control
Suyi Gao, Mo Zhou, Rongjie Lai
Stochastic mean-field control (MFC) provides a fundamental framework for coordinating large populations of interacting agents under uncertainty, with a wide range of applications.…
Training-Free Universal Approximation by Prompting Random Transformers
Alexander Hsu, Rongjie Lai
How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for…
Neural Operator Learning for Collision-Aware Trajectory Planning of Spacecraft Swarms
Sidhdharth D. Sikka, Suyi Gao, Zehui Lu +2
Satellite constellations require orbital transfers that are both fuel efficient and collision avoidant. Yet, the computational cost of optimization methods traditionally used to pl…
Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
Alexander Hsu, Zhaiming Shen, Wenjing Liao +1
Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite…
In-Context Operator Learning on the Space of Probability Measures
Frank Cole, Dixi Wang, Yineng Chen +2
We introduce \emph{in-context operator learning on probability measure spaces} for optimal transport (OT). The goal is to learn a single solution operator that maps a pair of distr…
Learn to Evolve: Self-supervised Neural JKO Operator for Wasserstein Gradient Flow
Xue Feng, Li Wang, Deanna Needell +1
The Jordan-Kinderlehrer-Otto (JKO) scheme provides a stable variational framework for computing Wasserstein gradient flows, but its practical use is often limited by the high compu…
Self-Supervised Amortized Neural Operators for Optimal Control: Scaling Laws and Applications
Wuzhe Xu, Jiequn Han, Rongjie Lai
Optimal control provides a principled framework for transforming dynamical system models into intelligent decision-making, yet classical computational approaches are often too expe…
Data-Driven Model Reduction using WeldNet: Windowed Encoders for Learning Dynamics
Biraj Dahal, Jiahui Cheng, Hao Liu +2
Many problems in science and engineering involve time-dependent, high dimensional datasets arising from complex physical processes, which are costly to simulate. In this work, we p…
Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods
Zhaiming Shen, Alexander Hsu, Rongjie Lai +1
While in-context learning (ICL) has achieved remarkable success in natural language and vision domains, its theoretical understanding-particularly in the context of structured geom…
Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights
Zhaiming Shen, Alex Havrilla, Rongjie Lai +2
Transformers serve as the foundational architecture for large language and video generation models, such as GPT, BERT, SORA and their successors. Empirical studies have demonstrate…
Improving Clean Accuracy via a Tangent-Space Perspective on Adversarial Training
Bongsoo Yi, Rongjie Lai, Yao Li
Adversarial training has proven effective in improving the robustness of deep neural networks against adversarial attacks. However, this enhanced robustness often comes at the cost…
Emulating Complex Synapses Using Interlinked Proton Conductors
Lifu Zhang, Ji-An Li, Yang Hu +4
In terms of energy efficiency and computational speed, neuromorphic electronics based on non-volatile memory devices is expected to be one of most promising hardware candidates for…
Generalization Error Guaranteed Auto-Encoder-Based Nonlinear Model Reduction for Operator Learning
Hao Liu, Biraj Dahal, Rongjie Lai +1
Many physical processes in science and engineering are naturally represented by operators between infinite-dimensional function spaces. The problem of operator learning, in this co…