Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models

Steve Yadlowsky, Lyric Doshi, Nilesh Tripuraneni

Transformer models, notably large language models (LLMs), have the remarkable ability to perform in-context learning (ICL) -- to perform new tasks when prompted with unseen input-o…