Transformers Don't In-Context Learn Least Squares Regression

Joshua Hill, Benjamin Eyre, Elliot Creager

In-context learning (ICL) has emerged as a powerful capability of large pretrained transformers, enabling them to solve new tasks implicit in example input-output pairs without any…