Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers

Hao Yan, Haolin Yang, Yiqiao Zhong

Transformers are effective at inferring the latent task from context via two inference modes: recognizing a task seen during training, and adapting to a novel one. Recent interpret…