Transformers are almost optimal metalearners for linear classification

Roey Magen, Gal Vardi

Transformers have demonstrated impressive in-context learning (ICL) capabilities, raising the question of whether they can serve as metalearners that adapt to new tasks using only…