Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning

Haolin Yang, Hakaze Cho, Yiqiao Zhong +1

The unusual properties of in-context learning (ICL) have prompted investigations into the internal mechanisms of large language models. Prior work typically focuses on either speci…