2 papers
cs.LG2025
Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions
Yanna Ding, Songtao Lu, Yingdong Lu +2
Transformer architectures can solve unseen tasks based on input-output pairs in a given prompt due to in-context learning (ICL). Existing theoretical studies on ICL have mainly foc…
math.DS2024
Dynamical Behaviors of the Gradient Flows for In-Context Learning
Songtao Lu, Yingdong Lu, Tomasz Nowicki
We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geom…