11 citations · 14 across the 6 of their papers we have counts for
6 papers
Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
Yingcong Li, Ankit Singh Rawat, Samet Oymak
Recent research has shown that Transformers with linear attention are capable of in-context learning (ICL) by implementing a linear estimator through gradient descent steps. Howeve…
Mechanics of Next Token Prediction with Self-Attention
Yingcong Li, Yixiao Huang, M. Emrullah Ildiz +2
Transformer-based language models are trained on large datasets to predict the next token given an input sequence. Despite this simple training objective, they have led to revoluti…
From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
M. Emrullah Ildiz, Yixiao Huang, Yingcong Li +2
Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation. In this work, we study learning a 1-layer…
Provable Pathways: Learning Multiple Tasks over Multiple Paths
Yingcong Li, Samet Oymak
Constructing useful representations across a large number of tasks is a key requirement for sample-efficient intelligent systems. A traditional idea in multitask learning (MTL) is…
Transformers as Algorithms: Generalization and Stability in In-context Learning
Yingcong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos +1
In-context learning (ICL) is a type of prompting where a transformer model operates on a sequence of (input, output) examples and performs inference on-the-fly. In this work, we fo…
Stochastic Contextual Bandits with Long Horizon Rewards
Yuzhen Qin, Yingcong Li, Fabio Pasqualetti +2
The growing interest in complex decision-making and language modeling problems highlights the importance of sample-efficient learning over very long horizons. This work takes a ste…