Optimal In-context Adaptivity and Distributional Robustness of Transformers

Tianyi Ma, Tengyao Wang, Richard J. Samworth

We study in-context learning problems where a Transformer is pretrained on tasks drawn from a mixture distribution $\pi ={\sum }_{\alpha \in \mathcal{A}}{\lambda }_{\alpha }{\pi }_{\alpha }$, called the pretraining prior,…